Why DORA metrics alone are insufficient?

Consider a world where metrics and dashboards do not exist, where your work is free from constraints and you have the freedom to explore your imagination, creativity, and innovative ideas without being tethered to anything.

It may sound like a utopian vision that anyone would crave, right? However, it is not a sentiment shared by business owners and managers who operate in a world where performance is defined by OKRs, KPIs, and accountability. In this environment, dreaming and fairy tales have no place.

Given that distributed teams are becoming more prevalent and the demand for rapid development is skyrocketing, managers are seeking ways to maintain control. Managers have recently favored using “DORA metrics” as a means of achieving this goal in the context of development teams. By tracking and trying to enhance these metrics, managers feel as though they have some degree of authority over their engineering team’s performance and culture.

However, here’s a message for all the managers out there on behalf of developers: DORA metrics alone are insufficient and won’t provide you with the help you require.

What are DORA metrics?

Before we understand, why DORA is insufficient today, let’s understand what are they! 

The widely used reference book for engineering leaders called Accelerate introduced the DevOps Research and Assessment (DORA) group’s four metrics, known as the DORA 4 metrics. 

These metrics were developed to assist engineering teams in determining two things: A) The characteristics of a top-performing team, and B) How their performance compares to the rest of the industry.

The four metrics are as follows: 

  • Deployment Frequency
  • Cycle Time (also known as Lead Time for Changes)
  • Mean Time to Restore (also known as Time to Restore Service).
  • Change Failure Rate

In their words:

“Deployment Frequency and Lead Time for Changes measure velocity, while Change Failure Rate and Time to Restore Service measure stability. And by measuring these values, and continuously iterating to improve on them, a team can achieve significantly better business outcomes.” 

Below are the performance metrics categorized in Elite, High, Medium & Low for 4 metrics – 

Why you’ve been using DORA wrongly?

Relying solely on DORA metrics to evaluate team performance has limited value. Leaders must now move beyond these metrics, identify patterns, and obtain a comprehensive understanding of all factors that impact the software development life cycle (SDLC). 

For example, if a team’s cycle time varies and exceeds three days, while all other metrics remain constant, managers must investigate deployment issues, the time it takes for pull requests to be approved, the review process, or a decrease in a developer’s productivity. 

If a developer is not coding as many days, what is the reason behind this? Is it due to technical debt, frequent switching between tasks, or some other factor that hasn’t yet been identified? Therefore, leaders need to look beyond the DORA metrics and understand the underlying reasons behind any deviations or trends in performance.

For DORA to produce reliable results, it is crucial for the team to have a clear understanding of the metrics they are using and why they are using them. While DORA can provide similar results for teams with similar deployment patterns, it is essential to use the data to advance the team’s performance rather than simply relying on the numbers. DORA should be used in combination with other engineering analytics to gain a complete picture of the development process, which includes identifying bottlenecks and areas for improvement.

However, poor interpretation of DORA data can occur due to the lack of uniformity in defining failure, which is a challenge for metrics like CFR and MTTR. Using custom information to interpret the results is often ineffective. Additionally, DORA metrics only focus on velocity and stability, and do not consider other factors such as the quality of work, productivity of developers, and the impact on the end-user. Therefore, it is important to use other indexes for a proactive response, qualitative analysis of workflows, and SDLC predictability to gain a 360-degree profiling of the team’s workflow.

To achieve business goals, it is essential to correlate DORA data with other critical indicators like review time, code churn, maker time, PR size, and more. Using DORA in combination with more context, customization, and traceability can offer a true picture of the team’s performance and identify the steps needed to resolve bottlenecks and hidden fault lines at all levels. Ultimately, DORA should be used as a tool for continuous improvement, product management, and enhancing value delivery.

Conclusion

While DORA serves its purpose well, it is only the beginning of improving engineering excellence. Looking at numbers alone is not sufficient; engineering managers should also focus on the practices and people behind the numbers and the barriers they face to achieve their best. It is a known fact that engineering excellence is directly related to a team’s productivity and well-being. Therefore, it is crucial to consider all factors that impact a team’s performance and take appropriate steps to address them.