“Sorry we don’t have enough resources, we only have four pairs” – As an engineering leader no other statement has made me cringe more. After all four pairs is a healthy sized team of eight developers.
Throughout my career I have run across CTOs, VPs, directors, development managers, teams, and individual developers who swear by pair programming with near religious devotion. Personally I’ve maintained a healthy dose of skepticism when it comes to pairing as an overarching development philosophy.
As an engineering leader my job is to build products that delight customers in the most efficient way possible. Anecdotally, pairing consistently costs more and hence seems irresponsible to use exclusively as a development technique. But admittedly anecdotal evidence is insufficient so I decided to dig through the research and see if I could find more empirical evidence to support my claim.
Background
Pair programming is an agile software development methodology where two programmers work on the same task using one computer and keyboard. One programmer is called the driver and operates the keyboard and does the primary coding work. The other developer, often called the navigator, is responsible for observing the driver and providing guidance in order to speed up problem solving, improve design, and minimize defects.
The potential negative impact of pair programming is immediately clear to most people. By applying two resources to a task you are effectively doubling the cost. So unless there’s an equal or greater improvement in other project variables, pair programming would be nearly impossible to justify. Exploring the problem through a project management lens, where we have three variables, cost (including resources), time, and quality/scope, If we double our cost we’d expect to see an equivalent decrease in time to deliver or increase in quality or scope (or some factor of each).
In mathematical terms let’s assume the value of any given project X is equal to a weighted linear combination of cost, time and quality/scope.
When pairing our cost is automatically going to double since we’ve applied two resources for a task that in theory can be completed by one.
In order for our project value to remain equal or be better we need our other variables to proportionally change in the right direction. For example if our project now takes 50% less time we could argue we net out even. Or if our scope or quality double, we would similarly be in a good position.
However, In my experience I’ve not seen pair programming live up to these expectations. Instead I’ve seen tasks or user stories take the same amount of time and produce similar results at nearly double the cost. But you shouldn’t take my word for it. Let’s review the literature and see what the experts have to say.
Research
There are actually a fair number of research papers that attempt to prove or disprove the efficacy of pair programming. That said, in my survey of the literature I found most of the research to be ill designed for comparison to real world corporate product development organizations. Specific issues include:
In spite of these issues it’s worth exploring these various research studies and the insights they provide on the impacts of pair programming.
Many of the research papers evaluate the impact of pair programming on effort, which in at least one paper is defined as two times the duration or time required to complete a given task [1]. Specifically, effort increases ranging from 15% all the way to 100% have been observed [2]. In one of the more well conducted studies an effort increase of 84% was seen [1]. Since we know effort is just twice the duration of a single developer we can actually do some math to figure out how much faster pairs complete a task versus a single developer.
Or by using our earlier project management equation, with a little rounding we can assume our pairing time weight would be roughly 9/10 the weight required for a single developer.
This is nowhere near the factor of 1/2 or less we said we needed to make pair programming cost efficient. Well if the research doesn’t support a sufficient decrease in time to completion perhaps there’s research indicating that a given project’s scope or quality will increase enough to offset the difference.
Unfortunately, once again the results are at best inconclusive, but in many cases support an actual decrease in scope and minimal or near zero increase in quality. For example in [2] a reported 29% decrease in productivity was measured for pair programming team when measured as a function of completed use cases.
Regarding quality, even in one of the more optimistic papers we only saw a 10% – 20% increase in quality (measured as test cases passed) [3]. According to [2], we only saw an 8% improvement in quality when measuring actual defects. While these improvements are non trivial, when combined with the time and scope metrics it remains insufficient to offset the associated costs.
Cherry Picking
“But aren’t you just cherry picking the worst examples to justify your case” you might ask? Not really because even in the most optimistic research studies initial results were usually much worse and only improved over time. For example in [3] initial increases in effort dropped from 60% to15% over time. Most of the research attributes these gains in effort to “pair jelling”. In other words, as the pairs get to known each other they become more efficient.
The problem with these studies is that they assume that once a pair jells the gain will hold. However in any real for-profit organization there is potential for high variability in projects and staff which means pair jelling is unlikely to be a one off cost. It is more likely a continuing cost to the business over time.
Several studies also point out that the value of pair programming decreases with simpler tasks [4]. Therefor one must consider the ratio of simple to complex tasks in any given development cycle in order to understand the long term impacts of pair programming. When I evaluated my own teams, I found multiple iterations where 75% of work items where smaller changes that could easily be tackled by a single developer in the same timeframe.
Finally, one paper [5] attempted to justify pair programming by evaluating Net Present Value (NPV). In this paper an argument is made that even if it costs more to pair program, faster time to market warrants the cost. I take issue with this calculation since it does not factor in the opportunity cost of having those extra resources not work on a different higher priority project.
For example if we take the reported 84% increase in effort and assume we finish our project in 9/10 the time of a single developer, we must ask ourselves what happens when a key customer asks for a critical bug fix? I can tell that customer to wait until I finish my current project or I can split my pair and work on both at the same time at the small cost of a 1/10 increase in duration. By splitting my pair I’ve delighted my key customer as quickly as possible at a trivial cost. Clearly you need to factor in the opportunity cost of not delighting that customer when evaluating the value of pair programming.
To Pair or Not to Pair
So should you pair or not pair? There are a lot of reasons a team might use pair programming. In some cases the cost / benefit tradeoff may be worthwhile. Pairing can be very effective at educating new team members, improving the skills of junior team members, cross training, and reducing the cost of complex tasks. If you take anything away from this post let it be:
In short don’t allow yourself to be swayed by a dogmatic insistence that pair programming is better. As a leader your job is to challenge your team to delight customers in the most cost effective way possible. Pairing should only be used if it definitively contributes to that cause.
References
[1] Arisholm, Erik, et al. “Evaluating pair programming with respect to system complexity and programmer expertise.” IEEE Transactions on Software Engineering 33.2 (2007). – Summary available at https://pdfs.semanticscholar.org/9787/c9663cad3a1c21550f2e5e365e70fd01d3aa.pdf
[2] Vanhanen, Jari, and Casper Lassenius. “Effects of pair programming at the development team level: an experiment.” Empirical Software Engineering, 2005. 2005 International Symposium on. IEEE, 2005. https://pdfs.semanticscholar.org/40dd/fa666bf367cfffaae421dbd3c6170a3e3dc3.pdf
[3] Cockburn, Alistair, and Laurie Williams. “The costs and benefits of pair programming.” Extreme programming examined (2000): 223-247. http://www.cs.pomona.edu/~markk/cs121.f07/supp/williams_prpgm.pdf
[4] Lui, Kim, and Keith Chan. “When does a pair outperform two individuals?.” Extreme programming and agile processes in software engineering (2003): 1011-1011. ftp://nozdr.ru/biblio/kolxo3/Cs/CsLn/E/Extreme%20Programming%20and%20Agile%20Processes%20in%20Software%20Engineering,%204%20conf.,%20XP%202003(LNCS2675,%20Springer,%202003)(ISBN%203540402152)(479s)_CsLn_.pdf#page=240
[5] Padberg, Frank, and Matthias M. Muller. “Analyzing the cost and benefit of pair programming.” Software Metrics Symposium, 2003. Proceedings. Ninth International. IEEE, 2003. http://wwwipd.ira.uka.de/Tichy/uploads/publikationen/32/metrics03.pdf