Mildly Interesting

A critique of the NYTimes article about Google’s big data study on interviewing

Posted on June 21, 2013 by Pankaj Gupta

This New York Times article has been doing the rounds on Twitter recently. Titled “In Head-Hunting, Big Data May Not Be Such a Big Deal”, it is a condensed and edited interview of Laszlo Brock, an SVP of people operations at Google. It is apparent to me that the title as well as the apparent nuggets of wisdom present in this article form just the sort of potent combination that invites endless tweets and retweets from the San Francisco/Silicon Valley echo chamber. Well done, Adam Bryant.

However, I find a few too many issues with this article that I am compelled to note. An article mentioning a big data study at a company as venerated for big data as Google naturally leads to wide-ranging conclusions. As I explain below, that would be wrong.

First of all, I read the article three times, but could not find anything emphatic in the body of the article that warrants the rather strong claim in the title that “Big Data may not be such a big deal”. On the contrary, Brock clearly says that just giving managers and leaders a visible set of measurements was enough for them to strive to improve. Yes, there is mention of the (organizational) context being important and the need for human insight, but even the most ardent fans of Big Data have never been heard to advocate jettisoning those. In another place, the article says “I think this will be a constraint to how big the data can get”. I can not help but wonder that the title might have been bolted on by some “social media expert” at NYTimes.

Then there are other questions the article and the correlation study mentioned raises.

Brock mentions that “the proportion of people without any college education at Google has increased over time as well. So we have teams where you have 14 percent of the team made up of people who’ve never gone to college.” Elsewhere, he says that the average team size is 6. Putting the two together, it means that 14% out of 6 i.e., less than 1 of the average team is made up of people who have not gone to college. In other words, Brock is very likely talking about bigger teams. It is not mentioned whether those teams are all engineering/product or contain other departments including some like support which apparently form a large part of Google? Or how many such teams are there? It is a bizarre piece of data mentioned by the article without saying how many such teams might be there.

The article cites a study at Google to detect correlation between interview types and success at Google after 2-3 years. It concludes that things like GPA, test scores, puzzles, brain teasers have no correlation and the only thing that correlates well is “structured behavioral interviewing”. At the very least, this deserves more explanation. Was the study done over all of Google, or only engineering or some other parts of the organization? Does the lack of correlation hold if you restrict to either top 20% or bottom 20%? Should everyone basically just do “structured behavioral interviews”? Now, I am myself a fan of behavioral interviewing (btw, here is a link if you want to read some examples of behavioral interview questions) and include a few such questions in my interviews of candidates at Twitter, but I can’t imagine the whole interview panel of an engineering or product position being of this sort.

Turning to the article’s juicy assertion that GPAs are not predictive of future success at Google, consider this from the article: “Google famously used to ask everyone for a transcript and G.P.A.’s and test scores, but we don’t anymore, unless you’re just a few years out of school.” So, Google still does ask those if you are a “just a few years out of school”? And if a candidate is experienced, I don’t know of anyone in Silicon Valley at least who would go with transcripts and GPAs, so the fact that Google also doesn’t is odd to mention.

My biggest gripe with the article is that since GPAs and puzzles are apparently not predictive in their study, what it leaves us at best is that we are back to square one. We are not told what is predictive so what are we to rely on? Lot of people would jump to the conclusion to not include those in an interview at all. But doing that would be wrong simply because who is to say whatever they would be replaced with will be more predictive…

To be sure, I am myself not a fan of going by GPAs and performance on brainteasers in interviews. I am not even casting doubts on that study at Google and in fact the results of the study look quite plausible to me, simply because in Data Analysis, not finding predictive variables is rather common. I am just distressed at the suggestive tone of the article. It invites wide-ranging prescriptions for how to and how not to interview. New York Times: I am used to better from you.

——

PS: BTW, in my own constant search for the answer to the question “what sort of interview should I do”, here is a recent book by Nolan Bushnell (the founder of Atari and supposed father of the video-game industry, and apparently someone who launched the career of Steve Jobs). Though it seems a bit as if Bushnell might be trying to ride the Steve Jobs’ popularity wave, it still presents many out-of-the-box ideas on ways to go about interviewing, especially if you are looking for creative and exceptional engineers (as I am at Twitter).

A note to engineering managers/management everywhere on the value of opening up the work of your teams

Posted on July 17, 2012 by Pankaj Gupta

Thanks to conferences and candidate interviews for recruiting, I find myself talking to a fair number of folks from other tech companies who are either engineers or in engineering management. Last week, a candidate was talking about how they did a particular project at his company (a mature public tech company). It was then that it suddenly occurred to me that I had heard similar stories many times before.

I now realize that there is a rather pervasive project pattern out there – that of coming up with what can only be called “yet another solution” to a commonly occurring problem, but never talking openly about the chosen path. Some examples of what I would call “commonly occurring problems” are: a spell-checker, typeahead on a search engine, reverse-proxy, load balancer, a recommendation engine, a method for generation of unique ids in a sharded store, etc. I have no issues at all with coming up with one’s own proprietary, home-baked solutions – it is indeed the case that company situations, availability of technologies and product needs are subtly different enough to often require a diversity of solutions.

At first, just the presence of endless varieties of home-baked solutions to the same problem used to baffle me. I now admit that this might be just because it stands in contrast to our software stack at Twitter, which, by virtue of our deep belief in the value of open-source, is heavily built on top of both external and internal open-source technologies.

Developing one’s own solutions is of course fine, but I have found more and more the pattern that whenever someone describes their proprietary technology or software no part of which has ever been made open in any form, it very quickly begins to sound like an unprincipled mish-mash of things hacked together because of company and project-specific circumstances. Worse, those circumstances often turn out to be such that the participant later finds it really difficult to articulate. So, what we end up with is that the work done was an unfortunate waste of time in the global sense of the advancement of the industry. Neither the design choices made, nor the lessons learned, nor any innovations made (small or big) are now available to be built further on. Worse for the company, lack of an unbiased source of review means that the work was likely sub-par.

I understand that there is need for “secret sauce” and proprietary algorithms and projects that can not be talked openly about. But, it is almost never the case in my experience that no part of the project is worth open-sourcing, or talking about in a workshop or publishing a blog post or paper about it. To give just a couple of examples, Twitter’s Revenue team developed and open-sourced Scalding, and the Search/Relevance team has openly talked about projects like earlybird and cassovary. All of these are key technologies from bigger, proprietary projects. Likewise, Facebook has open-sourced and/or published many of their key technologies (e.g., the “Learning Relevance” SIGIR paper that describes a key mechanism it uses in ad-targeting, a variety of interesting stats about its friendship graph, and more). Same holds with LinkedIn and Google over the years.

Therefore, I am convinced that if a project has not publicly opened any part of their work, the quality of work is very likely inferior. This is less damning a verdict on the engineers themselves than on their management chain. More often that not, it is the engineering management that is prone to think that doing the work needed (polishing, documenting, referencing previous work, explaining design choices etc.) is a waste of engineering time and without benefit to the company. This is plainly wrong. Openly talking about your work puts it under scrutiny and forces you to step up your quality and strive for being the best globally. Otherwise, it is too easy for a team to believe that it is under some unique circumstances and constraints such that its solution needed to be uniquely the way it ended up being implemented by the team. Promising yourself to make your choices and solutions open are a sure shot way to guard against this trap.

Personally, I emphasize to the engineers in my team that they absolutely need to open up their work – whether via open-sourcing, publishing a paper, or simply giving a company-wide tech-talk if their work is too sensitive to be published outside the company. It will be great if engineering management everywhere makes this a priority and an integral part of their own responsibility. I am convinced that doing so will help the individual, the company as well as the relevant industry as a whole.

Groupthink (and Centralized Bathrooms with Knowledge Spillovers)

Posted on January 30, 2012 by Pankaj Gupta

Two recent articles have appeared in respected publications on Groupthink: first is an NYTimes op-ed piece by Susan Cain on January 15th, and second is an article by Jonah Lehrer in the January 30th issue of New Yorker. (see here for a scanned PDF of the full article if you are not a subscriber.) Both authors have upcoming books on group creativity[4,5]. I was intrigued enough to do further readings based on the works cited in these articles. This blog post summarizes and links to those works for anyone interested in digging deeper.

[As an aside, after reading the underlying papers and studies cited in these articles, I was actually disappointed with the liberal oversimplifications that appeared to have been made by these pop writers, routinely papering over the disclaimers and conditions mentioned in the original papers. But I will leave that for another blog post.]

Jonah Lehrer in the New Yorker

The traditional brainstorming mantra prescribes forgoing judgment and criticism during a brainstorming session so as to maximise the quantity of free-flowing ideas and to not get any distractions from judgment. The word “brainstorm” and this mantra was put forth by Alex Osborn in his 1948 book “Your Creative Power.” However, it never stood up to empirical observations! Keith Sawyer, a psychologist at Washington University, is cited by Lehrer as summarizing the findings thus: “Decades of research have consistently shown that brainstorming groups think of far fewer ideas than the same number of people who work alone and later pool their ideas.” Ouch!

Lehrer does not advocate solitary work though. The fact is that it takes teams to produce substantive works. He refers to a study by Prof. Ben Jones which observed that the impact (quantified by the number of citations) of multi-author papers and patents (and especially across Tier-1 universities) are much higher than those of single author ones.

How should then groups maximize creativity?

To develop the answer, Lehrer cites the studies of psychology Prof. Charlan Nemeth at Berkeley and sociologist Brian Uzzi at Northwestern.

Nemeth observes that the groups doing traditional brainstorming consistently underperformed those groups that were given the instruction to engage in debate during brainstorming. Lehrer quotes her: “Maybe debate is going to be less pleasant, but it will always be more productive.” Also “Authentic dissent can be difficult, but it’s always invigorating. It wakes us right up.” I personally observed the waking up part recently after receiving some unexpected but deserved criticism from a respected colleague.

Uzzi has studied for a long time the ideal composition of a team with respect to “Q” – a quantification by him of a team’s existing density of connections and mutual familiarity. Studying the composition of Broadway musical teams, he found that having worked together correlates positively with the success of the musical. But he also found rather surprisingly that a team being too familiar with each other was bad for its work! In other words, the best shows were produced by networks with an intermediate level of social intimacy. Lehrer says: “A team at the bliss point i.e., with the ideal level of Q between 2.4 and 2.6 was 3x more likely to be successful than one with Q lower than 1.4 or higher than 3.2.”

Centralized Bathrooms and Knowledge Spillovers

Lehrer then turns his attention to organization of physical space to maximize productivity. He cites a study by Isaac Kohane from Harvard Medical school on physical distance between collaborating biomedical researchers with in Harvard. Kohane observed that the most cited papers were consistently by collaborators working within ten metres of each other, while the least cited ones tended to emerge from collaborators who were a kilometre or more apart. [This study was not done on a global framework, so should not mislead us to think that across-university collaborations are less impactful. On the contrary, this study by Ben Jones indicates that across-university collaborations have been consistently on the rise and much more impactful than solo. Disappointingly, Lehrer omits the disclaimers in Kohane’s original paper as well as the across-university collaborative research.]

So there are two needs that need to be reconciled: (1) collaborators need to work physically closely together, and (2) collaborators need to get diverse perspectives and inject in themselves a moderate amount of novel perspectives. Lehrer talks about Steve Jobs’s deliberate attempts to create such an environment in Pixar that encouraged chance interactions of people from diverse teams with each other, causing various opportunities for “Knowledge Spillovers.” Finally, Lehrer talks about “Building 20” at MIT that was referred to as “the magical incubator”. Apparently, this building was under-designed and happened to be so “poorly structured” that scientists were forced to mingle. For example, the wings were oddly ordered, a large horizontal layout encouraged more chance interactions and individual scientists were free to reorganize their space as they needed. The lesson, Lehrer says, is that “the most creative spaces are those that hurl us together. It is the human friction that makes the sparks.”

Susan Cain in the New York Times

Susain Cain writes about “The Rise of the New Groupthink” in the January 13th op-ed. She first cites studies by psychologists Mihaly Csikszentmihalyi and Gregory Feist (and quotes many historical famous personalities) according to whom the most spectacularly creative people in many fields are often introverted, and that people produce more and better quality work when alone than in groups. Note that this is in direct contrast to what Keith Sawyer has said that innovation comes from collaboration in his book “Group Genius.” Interestingly, Lehrer cited Sawyer while debunking the traditional theory of brainstorming.

About office plans, she writes: “Studies show that open-plan offices make workers hostile, insecure and distracted. They’re also more likely to suffer from high blood pressure, stress, the flu and exhaustion. And people whose work is interrupted make 50 percent more mistakes and take twice as long to finish it.”

Similar to Lehrer, Cain writes that brainstorming doesn’t work, partly because people instinctively or unconsciously mimic each other. She even cites an Emory University neuroscientist Gregory Berns who found that when we take a stance different from the group’s, we activate the amygdala, a small organ in the brain associated with the fear of rejection.

However, according to Cain, electronic brainstorming works well – perhaps “because the screen protects us from too much groupthink.” This is surprising and counterintuitive to me.

Cain clarifies that she is not suggesting that teams should be abolished and everybody should be a loner, but that office spaces need to be designed such that people can have “casual, cafe like discussions but also be able to easily disappear into personal and private spaces”.

In my humble opinion, Cain might be mixing up individual and team’s productivity and creativity. Indeed, individuals are most productive when working in solitude, more so when we are talking about exceptional geniuses. However, many studies have shown by now that collaborative groups can do far more and better work than individuals working together in isolation. An organization is after all not the sum of each individual’s achievements, it is the sum of each team’s achievements. I guess Cain’s book [4] throws more light into her argument.