Can developer productivity be measured? Better than you think

by Chandra Gnanasambandam and Martin Harrysson

Opinion

Nov 20, 20235 mins

ROI and MetricsSoftware Development

After decades of debate over whether developer efficacy can truly be measured, we believe we’ve found the developer productivity metrics that matter most. Here’s why it matters.

Measuring developer productivity has long been a Holy Grail of business. And like the Holy Grail, it has been elusive. But based on our work with companies from a range of industries, we think we may have figured out a way to do it that could work.

In 2020, McKinsey surveyed 440 large companies about their “developer velocity” — meaning the practices that best tap the full potential of development talent. The results were striking. Companies in the top quarter delivered four to five times faster revenue growth than those in the bottom quarter. The high performers also saw 60% higher shareholder returns and 20% higher operating margins. Their customers were more satisfied, and their business colleagues had a better employee experience.

And this doesn’t matter only for tech companies. In retail, for example, software has been the fastest-growing job category; about half of the world’s software engineers work outside the tech industry. Right now, there are roughly 27 million developers on the job, 4.4 million in the United States. The US Bureau of Labor Statistics has projected that the number of software developers will grow 25% from 2021-31. Given the rise of generative artificial intelligence, that could well be a massive underestimate.

All this data leads to a simple conclusion: Leaders need to know they are deploying developer talent in the best way possible. That isn’t easy. The relationship between inputs and outputs is murky, and software development is inherently collaborative and creative. In addition, system, team, and individual productivity all need to be measured. Well-known metrics, such as deployment frequency, are useful when it comes to tracking teams but not individuals. So, it’s complicated. But we believe it can be done.

The developer productivity metrics that matter most

The reason we believe this is that we are working with 20 tech, finance, and pharmaceutical companies that are doing it. The results are not definitive, but they are promising. Based on internal research, when these companies acted based on the following process, they recorded positive results related to customer defects (20% to 30% down); employee experience (20% up); and customer satisfaction (60% up).

Here’s how it works. We started with two established sets of metrics, developed by Google and Microsoft, respectively: DORA (an acronym for DevOps Research and Assessment team), which measures outcomes; and SPACE (an acronym for satisfaction, performance, activity, communication/collaboration, and efficiency), which is good at evaluating measures related to optimization, such as interruptions. Then we complemented these with the following four “opportunity-focused metrics.”

Inner/outer loop time spent. The inner loop comprises activities directly related to creating the software product: coding, building, and unit testing. The outer loop comprises activities related to putting the code into production: integration, testing, release, and deployment. When developers spend more of their time in the inner loop, they are more productive; at top performers, this is about 70%.

Developer velocity index benchmarking. By comparing a company’s practices against its peers, it is possible to unearth specific areas to improve, whether in backlog management, testing, or security and compliance. Greater maturity in development practices is associated with better company performance.

Contribution analysis. This refers to assessing contributions to a team’s backlog. Using tools such as Jira, which measures backlog management, it is possible to spot trends that are damaging to optimization. The process can also reveal opportunities, such as improving the working environment, increasing automation, or enhancing individual skills, to fix problems that can hurt performance. One business, for example, found that the developers who were making the greatest contributions were spending too much time on noncoding activities. The company changed its operating model to ensure that they focused on what they did best.

Talent capability. The idea here is to be sure that the right people are in the right place. By deploying industry-standard capability maps, it is possible to create a score that summarizes the individual knowledge, skills, and abilities of a specific organization. This can reveal both gaps and bulges. For example, one company found it had a too many inexperienced developers. In response, it took action, including providing personalized learning journeys, and moved 30% of its developers to the next level of expertise within six months.

Combined with DORA and SHAPE, these tools effectively create a sophisticated view of software productivity. The insights revealed are intrinsically interesting. The value comes from using them to figure out how to keep developers motivated; whether they have the right tools and expertise; how they are using their time; and if staffing levels are correct.

Improving an imperfect model

Like the Holy Grail, there are those who think that measuring developer productivity is a myth and that we are off base. But the 20 companies that we are working with would disagree.

Moreover, we don’t accept that software engineering is so complicated or mystical that it defies measurement. Rather the opposite: McKinsey has been able to estimate the improvements associated with the use of generative AI-based tools in several areas, including drafting new code and implementing updates.

The system we have described is undoubtedly imperfect; indeed, we welcome criticism that can improve it. But given the ever-growing importance of software development, and the ever-fiercer competition for talent, it’s too important not to try.