Thanks to conferences and candidate interviews for recruiting, I find myself talking to a fair number of folks from other tech companies who are either engineers or in engineering management. Last week, a candidate was talking about how they did a particular project at his company (a mature public tech company). It was then that it suddenly occurred to me that I had heard similar stories many times before.

I now realize that there is a rather pervasive project pattern out there – that of coming up with what can only be called “yet another solution” to a commonly occurring problem, but never talking openly about the chosen path. Some examples of what I would call “commonly occurring problems” are: a spell-checker, typeahead on a search engine, reverse-proxy, load balancer, a recommendation engine, a method for generation of unique ids in a sharded store, etc. I have no issues at all with coming up with one’s own proprietary, home-baked solutions – it is indeed the case that company situations, availability of technologies and product needs are subtly different enough to often require a diversity of solutions.

At first, just the presence of endless varieties of home-baked solutions to the same problem used to baffle me. I now admit that this might be just because it stands in contrast to our software stack at Twitter, which, by virtue of our deep belief in the value of open-source, is heavily built on top of both external and internal open-source technologies.

Developing one’s own solutions is of course fine, but I have found more and more the pattern that whenever someone describes their proprietary technology or software no part of which has ever been made open in any form, it very quickly begins to sound like an unprincipled mish-mash of things hacked together because of company and project-specific circumstances. Worse, those circumstances often turn out to be such that the participant later finds it really difficult to articulate. So, what we end up with is that the work done was an unfortunate waste of time in the global sense of the advancement of the industry. Neither the design choices made, nor the lessons learned, nor any innovations made (small or big) are now available to be built further on. Worse for the company, lack of an unbiased source of review means that the work was likely sub-par.

I understand that there is need for “secret sauce” and proprietary algorithms and projects that can not be talked openly about. But, it is almost never the case in my experience that no part of the project is worth open-sourcing, or talking about in a workshop or publishing a blog post or paper about it. To give just a couple of examples, Twitter’s Revenue team developed and open-sourced Scalding, and the Search/Relevance team has openly talked about projects like earlybird and cassovary. All of these are key technologies from bigger, proprietary projects. Likewise, Facebook has open-sourced and/or published many of their key technologies (e.g., the “Learning Relevance” SIGIR paper that describes a key mechanism it uses in ad-targeting, a variety of interesting stats about its friendship graph, and more). Same holds with LinkedIn and Google over the years.

Therefore, I am convinced that if a project has not publicly opened any part of their work, the quality of work is very likely inferior. This is less damning a verdict on the engineers themselves than on their management chain. More often that not, it is the engineering management that is prone to think that doing the work needed (polishing, documenting, referencing previous work, explaining design choices etc.) is  a waste of engineering time and without benefit to the company. This is plainly wrong. Openly talking about your work puts it under scrutiny and forces you to step up your quality and strive for being the best globally. Otherwise, it is too easy for a team to believe that it is under some unique circumstances and constraints such that its solution needed to be uniquely the way it ended up being implemented by the team. Promising yourself to make your choices and solutions open are a sure shot way to guard against this trap.

Personally, I emphasize to the engineers in my team that they absolutely need to open up their work – whether via open-sourcing, publishing a paper, or simply giving a company-wide tech-talk if their work is too sensitive to be published outside the company. It will be great if engineering management everywhere makes this a priority and an integral part of their own responsibility. I am convinced that doing so will help the individual, the company as well as the relevant industry as a whole.