Disclosure: I recently worked for Google for about a year. It was alright.
Recently, Vic Gundotra of Google+ fame made a bold statement. He
that the lack of write API access to Google+ is born not out of lack of
foresight, planning, or even bandwidth, but out of trepidation, caution, and the
desire to do right by developers.
This is raw, barely-refined bullshit and I regret not being able to respond to
the thread with a pithier snarky comment. The truth is simpler: Google is
full-stop terrible at APIs.
Don’t look at me, it says so right there
As a case study, let’s examine what I had to do in order to use the most
powerful collection of user-generated content that Google makes available to
developers: a user’s email contact list. People who work for consumer-
facing web companies probably know how deeply important having a user’s email
address is for marketing. Despite being invented before I was born, email has
yet to be outmoded as the most effective way to push content to users on demand.
Recently here at Everlane, we thought it might be a good idea to have a
button a user could hit to view a list of their Gmail contacts with portraits,
select some of them, and then have us send those users an invitation email.
Simple, obvious stuff to want to do, right?
What we want to implement
Well, let’s hit the getting started page of the contacts API documentation.
Reading closely seems to reveal that there might already be a library for doing
what we want, which is really nothing more than getting a bunch of names, faces,
and emails. Those geniuses at Google have to have something already made for
this, right? We head over to the libraries and samples page. Cool, a
Google API but Contacts. The Google+ read API doesn’t seem to have a good way to
grab emails, either.
That’s fine – we’re pretty decent engineers and can call the Contacts API on
our own. We register our application with the API Console and start reading
about OAuth. Google provides a few authorization schemes. We probably want
the one titled “Client-side Applications”, which saves us the complexity of
an application server having to be aware of any sensitive information.
Now we can finally ask the Contacts API for a list of contacts! Easy enough,
right? We’ll just do a JSONP request to get around the cross-domain
What does this give us? To our slow and creeping horror, we get back something
like this for every contact:
You take several deep breaths. You ignore the fact that you are fetching roughly
1kb of data per contact (out of potentially thousands) to get a name, an email,
and the URL of an image. “Okay”, you think to yourself, “this is still
salvageable. I can parse XML on the client. In fact, jQuery can probably do it
for me.” You take a quick stab at grabbing names and emails.
A quick browser test shows that this only appears to work in Chrome. A bit more
digging turns up, to your chagrin, that jQuery has trouble finding namespaced
elements like “" in XML documents on different browsers. You Google
around and find a fix:
For now, you ignore how inefficient this is, hoping merely to reach
functionality. It works! Now you want to add images. It looks like one of the
link elements under <entry> appears to point to an image for that contact. You
fiddle around on the console:
Attempting to load this in your browser gives you a 401. Taking a look a the
photo management section of the docs seems to suggest you need to
additionally apply auth credentials to this url. You amend your code:
This seems to produce a real photo in your browser. Success! Let’s extrapolate:
Unfortunately, only a few images load. What’s going on? You squint at the docs
Note: If a contact does not have a photo, then the photo link element has no
Great, so some image links have a magic attribute on them that says they’re real
images. You wonder why Google even bothers returning image links for photos that
don’t exist. You try something like the following:
But no, jQuery can’t deal with namespaced attribute selection, at all, so you
This time, a few more photos load, but then they stop coming. The console shows
a few successful image loads, but most of the requests for images returned with
503 (Service Unavailable) errors. You realize, after an hour or so, that
each image load is being counted as an API call against you, and that there must
be some rate limiting in place.
Naturally, this fact is completely undocumented. Playing around with the
code, you find that Google doesn’t like it if you have more than one in-flight
API request at a time. You come up with essentially the opposite of an image
preloader to stagger image loading:
Whew, that was fun, right? At this point it seems like a good idea to try to
sort contacts by some sort of relevance metric. Unfortunately, the Contacts API
doesn’t support this at all. Oh well. You give up, having reached something
approximating your original goals.
What have we learned?
Google doesn’t get JSON.
Google can’t design clean APIs or document them well.
Despite their browser-first mantra, Google doesn’t put out first-party
Vic Gundotra is soooooo lying.
Developers waiting for Google+ to deliver on its full, API-wonderland potential:
you should probably just give up. What are the odds that this whole time,
they’ve been cooking up the perfect write API, replete with features, libraries,
and documentation? I’m betting that they’re doing exactly what I did when I
worked for Google: absolutely nothing of importance.
Lastly, we’re always on the lookout for talented developers who are interested
in the fashion space here at Everlane. You don’t need to own more than one
pair of shoes. My love of fashion is born of William Gibson novels and is almost
entirely academic. If you’re interested, check out our jobs page or send
me an email at firstname.lastname@example.org.
P.S. Yes, I am aware there is an older gdata library that can
potentially handle contacts. I might have used it if it wasn’t deprecated or
Google had made any mention of it whatsoever on their Contacts API page.
It was inspired by this very similar and honestly much better video.
I think I’ll play with attempting to get actually separate instrument tracks
instead of trying to muddle with frequency filters to try to split them.
This piece was written for an internal Google fiction contest, for the 100th
edition of the engineering newsletter. The call to arms arrived in my inbox like
For this special Eng Newsletter issue, we’re running a “google eng-y” short
fiction contest. You can write about anything, but the story must begin with
these two words: “The MapReduce”.
Please note that some meaning may be lost on non-Googler’s, notably the bits
concerning company hierarchy. All the opinions expressed are my own and
obviously do not constitute the workings of an actual Google plan, etc. Jeff Dean
is a very nice man. This is a piece of fiction in almost every sense.
The MapReduce was a piece of technology whose existence its steward, Jeff Dean,
sometimes begrudged. It was glamorous, in a way, to be the public face of the
algorithm that had essentially rewritten interpersonal contact, but it was also
draining and surreal.
In one of Jeff’s increasingly common attacks of perspective, he realized that
his daughters, too, had been completely swept up by a thing that he himself had
designed, built, and evangelized. They were, of course, perfectly happy with the
product. Jeff noted this with a tinge of grim pride, remembering the long nights
of trial runs. Victoria and Natalie were a bit too happy, Jeff mused, so
completely satisfied with something they could never understand (indeed, that he
himself no longer understood well), that he found their lack of doubt troubling.
Why didn’t they care that it probably shouldn’t work, that time and computation
could twist statistics in such a fundamentally disturbing way? It was probably
due to both of them being so preoccupied with Natalie’s wedding, he concluded
Later, outside of his office, Irina was waiting for him.
“Jeff, you have a visitor waiting for you in your office,” she said. Something
in her tone gave away the urgency of the situation, and Jeff nodded, having long
grown used to trusting Irina to manage his calendar more deftly than he could
tie his shoes.
His suited visitor was a trim man of about sixty, which was unusual enough for
the Googleplex in terms of both age and dress. He wore his graying hair swept
back and neatly cropped. With a start, Jeff realized that his visitor was none
other than a senator of Iowa.
“I’m Robert Graves, and sorry about showing up so unannounced, Mr. Dean,” said
the man, with a smile. Jeff paused for a moment to admire how finely
countenanced the man was, and to feel a small thrill at being so delightfully
“You’re the senator pushing for patent reform. I don’t watch TV much, but I’ve
seen you on when my wife watches the news.” Jeff shook his visitor’s hand and
seated himself behind his desk.
“The very same. Look, I’ll spare you the pleasantries and get right to why I’ve
come. I’m told that engineers prize truth and directness.” Jeff lifted an
eyebrow at this, having found that lately he valued being left well enough alone
better than both of those things. “As you well know, MapReduce is proving
problematic, socially. FOX is filming a reality TV show at this very moment
about an engaged couple who are convinced that after trying out their MapReduce
partners, they’ll still want to get married.”
“Jesus. How’s it looking for the couple?”
“Not good. Even worse, they’re filming it in my hometown.” Graves massaged his
Jeff was not surprised. MapReduce rarely erred. Though it had begun as a general
purpose framework for parallelizing search index updates, it eventually lent
itself to analyzing the massive amounts of user generated social data Google+
collected. In time, this would become all that MapReduce was known for (at least
externally of Google), in a queer reversal of how the words escalator and
aspirin came to describe all such contrivances, though they were once only
“Basically the right is getting as much fuel as it wants for its eternal fire of
shouting about our perpetual moral decay. On top of that, MapReduce is having a
powerful economic impact, which doesn’t help. We’re having an employment
problem, as you’ve doubtless inferred by now, since you must have all the
numbers on how many people are using MapReduce to pair up.”
The first Jeff had heard about the phenomenon the media had dubbed as the
“honeymoon effect,” had been from the news itself, but he nodded anyway. “My
citizens are up and leaving jobs they’ve worked at for a decade to meet their
dreamboat on the other side of the world. I mean, great for them, but our
coffers weren’t in great shape before, and your invention is a drain we can’t
possibly afford right now, never mind the bad press. As much as I am for the
future, I desperately need you to stop operating in my state.”
Jeff suppressed the urge to tell the man to just contact email@example.com, and
instead reluctantly launched into a narrative he had delivered many times
before. “I’m sure you’ve seen and read all the press releases about this. What
we do isn’t terribly new. We provide a service that users want. In a sense, we
provide nothing more than what eHarmony and Match.com have been providing for
years, just with much less uncertainty and a bigger candidate pool.”
Robert snorted. “I’d hardly call ‘every Google user on Earth’ a bigger pool.
Your operation is different, too. You know all the things that people have
searched for, and all the things they’re too ashamed to search for. You know why
some actresses draw men to them, and which men women will wait hours to receive
texts from. Those bankrupt dating sites had only the constructed personas of the
desperate to work with. There’s a case that could be made here for unlawful
invasion of privacy and monopolistic abuse of information.”
A Googler rode past Jeff’s window on a small yellow bicycle. Jeff focused on the
bright colors to briefly escape his current uncomfortable tension.
If Graves was right about anything, it was that MapReduce was uncannily
effective. Through what some people might call sorcery, or what Jeff’s team
leads described as “massively parallel Bayesian-adapted machine learning plus
deep social mining,” it was able to identify, with nearly 97% confidence, a
lifetime romantic partner for any given user. The algorithm could even supply
just the right amount of shared interests as conversation starters, while
leaving enough unsaid for the nascent couple to discover independently, leaving
them feeling as if they had come to know each other intimately of their own
volition. Some people found this deeply unnerving.
Even those people commonly derided by society could find love in this way,
though MapReduce might take weeks instead of seconds to produce a suitable
pairing. People of every sexual deviancy and every personal vice were being
matched up, to the horror of the many people alienated by the brutal efficiency
of MapReduce’s perfect lack of bias.
In short, romantic fulfillment was, for most people, little more work than
clicking “I’m feeling lucky” and buying a plane ticket. This is what the people
wanted more than anything else. Graves knew it, and Jeff knew that Graves knew
it. Furthermore, Jeff knew that Graves was powerless to do anything about it, so
strongly did the public crave MapReduce’s presence in the world. Yet Jeff felt
sympathy for Grave’s willingness to shoulder the impossible task of squaring the
budget against falling revenues and changing social tides.
“Mr. Graves, I understand your dilemma. The last thing you need right now is the
income rug pulled out from under you. But look at it this way: about half of
those people who have gone and paired off will probably come back to their
hometown, bride or husband in tow, so your population will probably end up about
even. After these couples outgrow their honeymoon period, they’ll settle down,
work, have kids, and spend with an intensity that only the truly content can
bring to bear. In the coming decade, your books might even make it into the
Robert was not easily placated. “Can you say for certain that this is the way
it’s going to play out? The world has never seen this kind of mass social
movement. What if the people become complacent instead of motivated? What if
your algorithms can’t guarantee long term stability?”
Jeff had an inner conflict. As usual, the side favoring the least amount of
social friction won out. “We’re the ones who managed to pair everyone up so well
in the first place, aren’t we? The models say the population will eventually
converge on a higher level of stable productivity. I can’t promise you it’s
going to happen, of course, but here at Google we have pretty high hopes for the
The two men talked in this way for some time. The elder statesman pushed and the
younger (but not exactly young) engineer deflected until the senator grew weary
or satisfied enough to defer discussion to a later date. Jeff had managed to end
the meeting with only vague promises, a surprising talent that had earned him
his relative autonomy from Larry Page’s inner circle. Later he would have to
file a report, naturally, detailing the intricacies of his conversation with the
senator, but for now Page trusted him to keep third parties at arm’s length on
Later that evening, after a quiet supper with Heidi, Jeff lay in bed thinking.
The models actually didn’t say much about the economic reality of the future.
The social data that allowed his team to pair people so effectively seemed to
shrug mutely at the problem of what the future might be like. He had assured
Graves that everything would be fine, but by the time Jeff could be proved
wrong, he would be long retired.
Sleep took him. He dreamt, which was not unusual (though he didn’t know it), but
he also remembered his dreams from that night, which was. He dreamt of a young
man smashing a perfect chalice in a decrepit hallway, and of women who laughed
while they danced away from their homes.
When he woke, Jeff knew what he had to do.
Sanjay probably could be trusted, but Jeff couldn’t take the chance. He would
split his change into pieces, and sneak them into other, tangentially related
changelists. The other developers on his team would probably rubber stamp these,
anyway, since Jeff was one of the most prolific programmers there was. Who would
look at yet another Jeff Dean code review too closely?
What did it mean to adhere to Google’s famed “Don’t be Evil” policy, when it
came to arranging marriages? The standard Google answer would be to make the
user as happy as possible without violating their trust. But what trust was
there to violate if users themselves didn’t know what they wanted in
relationships, or what would truly make them happy? Marriages are long lived
beasts, Jeff reasoned, subject to slowly building changes in the macroeconomic
climate. If marriages affect the economy, and the economy affects all marriages,
what should you optimize for, and how?
Jeff’s changelists were approved, as a matter of course. Years later, he
retired. The day he first started noticing what might have been the fruits of
his subversion ripen, he remembered a thing that his old mentor Urs loved to
say, before Urs had left him in charge.
“It is better to ask forgiveness than permission,” Hölzle would often chuckle,
in a particularly German way. Jeff chuckled now, too.
Wired was doing a bio piece on a recently minted tech millionaire. The man was
one of the few people for whom MapReduce’s pairing hadn’t worked out in the long
term. When asked what had motivated him to start the company he had just sold,
the man somewhat abashedly said that he wanted to prove to his ex-wife that
dumping him was a mistake.
Ambition and talent sometimes survive contact with love, Jeff mused, but are
more often dulled by it. MapReduce could identify those individuals who are
defined by intelligence, drive, and pride. In other words, the archetypal
entrepreneur. A few modified terms in a complex linear algebra equation could
yield surprising results, Jeff had discovered, like optimizing for romantic
partners that would net the largest increase in a person’s ambition, rather than
happiness. A lot of the unfortunate people of talent singled out by Jeff’s
modification would probably yield little value, but one, he hoped, would build
the next Google. Jeff longed to see that day.