In the fourth episode of ‘The DORA Lab’ - an exclusive podcast by groCTO, host Kovid Batra engages in an enlightening conversation with Peter Peret Lupo, Head of Software Engineering at Sibli, who brings over a decade of experience in engineering management.
The episode starts with Peter sharing his hobbies, followed by an in-depth discussion on how DORA metrics play a crucial role in transforming organizational culture and establishing a unified framework for measuring DevOps efficiency. He discusses fostering code collaboration through anonymous surveys and key indicators like code reviews. Peter also covers managing technical debt, the challenges of implementing metrics, and the timeline for adoption. He emphasizes the importance of context in analyzing teams based on metrics and advocates for a bottom-up approach.
Lastly, Peter concludes by emphasizing the significant impact that each team member has on engineering metrics. He encourages individual contributors and managers to monitor both their personal & team progress through these metrics.
Kovid Batra: Hi everyone. This is Kovid, back with another episode of our exclusive series, the DORA Lab, where we will be talking about all things DORA, engineering metrics, and their impact, and to make today's show really special, we have Peter with us, who is currently an engineering manager at Sibli. For a big part of his career, he has been a teacher at a university and then he moved into the career of engineering management and currently, holds more than 10 plus years of engineering management experience. He has his great expertise in setting up dev processes and implementing metrics, and that's why we have him on the show today. Welcome to the show, Peter.
Peter Peret Lupo: Thank you.
Kovid Batra: Quickly, Peter, uh, before we jump into DORA metrics, engineering metrics and dev processes, how it impacts the overall engineering efficiency, we would love to know a little bit more about you. What I have just spoken is more from your LinkedIn profile. So we don't know who the real Peter is. So if you could share something about yourself, your hobby or some important events of your life which define you today, I think that would be really great.
Peter Peret Lupo: Well, um, hobbies I have a few. I like playing games, computer, VR, sort of like different styles, different, uh, genres. Two things that I'm really passionate about are like playing and studying. So I do study a lot. I've been like taking like one hour every day almost to study new things. So it's always exciting to learn new stuff.
Kovid Batra: Great, great.
Peter Peret Lupo: I guess, a big nerd here!
Kovid Batra: Thank you so much. Yeah. No, I think that that's really, uh, what most software developers and engineering managers would be like, but good to know about you on that note.
Apart from that, uh, Peter, is there anything you really love or would you like to share any, uh, event from your life that you think is memorable and it defines you today who you are?
Peter Peret Lupo: Well, that's a deep question. Um, I don't know, I guess like, one thing that was like a big game changer for me was, uh, well, I'm Brazilian, I came to Canada, now I'm Canadian too. Um, so I came to Canada like six years ago, and, uh, it has been transformational, I think. Like cultural differences, a lot of interesting things. I feel more at home here, to be honest. Uh, but like, yeah, uh, meeting people from all over the world, it's been a great experience.
Kovid Batra: Great, great. All right, Peter. So I think, first of all, thanks a lot for that short, sweet intro about yourself. From this point, let's move on to our main topic of today, uh, which is around the engineering metrics and DORA metrics. Before we deep dive, I think the most important part is why DORA metrics or why engineering metrics, right? So I think let's start from there. Why these engineering metrics are important and why people should actually use it and in what situations?
Peter Peret Lupo: I think the DORA metrics are really important because it's kind of changing the culture of many organizations, like a lot of people were already into, uh, measuring. Measuring like performance of processes and all, but, uh, it was kind of like, sometimes it wasn't like very well seen that people were measuring processes and people took it personally and it's like all sort of things. But nowadays, people are more used to metrics. DORA metrics is like a very good framework for DevOps metrics, and so widespread nowadays, it's kind of like a common language, a common jargon, like when you talk about things like mean lead time for changes, everybody knows that, everybody knows how to calculate that. I guess that's like the first thing, like the changing the culture about measuring and measuring is really important because it allows you to, uh, to establish a baseline and compare the results of your changes to where you were before and, uh, affirm if you actually have improved, if something got worse with your changes, if your, the benefits of your changes are aligned with the organizational goals. It allows everybody to be engaged at some level to, uh, reaching the organizational goals.
Kovid Batra: Makes sense. Yeah, absolutely. I think when we always talk about these metrics, most of the people are talking about the first-level DORA metrics, which is your lead time for changes or cycle time, or the deployment frequency, change failure rate, mean time to restore. These metrics define a major part of how you should look at engineering efficiency as a manager, as a leader, or as a part of the team. But do you think is it sufficient enough? Like looking at just the DORA metrics, does it sound enough to actually look at a team's efficiency, engineering efficiency? Or do you think beyond DORA that we should look at metrics that could actually help teams identify other areas of engineering efficiency also?
Peter Peret Lupo: Well, um, one thing that I like about our metrics is that it lets us start the culture of measuring. However, I don't see that as like the only source of information, like the only set of metrics that matter. I think there are a lot of things that are not covered in DORA metrics. The way that I see, it's like it's a very good subset for DevOps, it covers many different aspects of DevOps, and that's important because when you wanna measure something, it's important to measure different aspects because if you are trying to improve something, you want to be able to detect like side effects that may be negative on other aspects. So it's important to have like a good framework. However, it's focused a lot on DevOps, and, uh, I'll tell you, like, if you are on a very large organization with a lot of developers pushing features, like many changes daily, and your goal is to be able to continuously deliver them and be able to roll back them and assess like the time to restore the service when something breaks down. That's good, that's very, very interesting. And so I think it's very aligned with like what Google does. Like it's a very big corporation, uh, with a lot of different teams. However, context matters, right? The organizational context matters. Not all companies are able, for instance, to do continuous delivery. And sometimes in our matter of like what the company wants or their capability, sometimes their clients don't want that, like if you have like banks as clients, they don't want you to be changing their production environments every like 12 hours or so. Uh, they want like big phases, uh, releases where they can like do their own testing, do their own validation sometimes. So it's fundamentally different.
In terms of, uh, the first part of it, because when you get to DevOps and you get to like delivery stuff into production, things were already built, right? So building is also something that you should be looking at. So DORA metrics provide a good entry point to start measuring, but you do need to look at things like quality, for instance, because if you're deploying something and you're rolling back, and I want to make a parenthesis there, if you're measuring deployment frequency, you should be telling those apart because rolling back a feature is not the same as, like, deploying a feature. But if you're rolling back because something wasn't built right, wasn't built correctly, there's a defect there. DORA metrics won't allow you to understand the nature of the defect, where you got into, like, got into, like the requirements and continue what's propagated to codes and tests, or if somebody made a mistake on the codes, like it doesn't allow you for this level of understanding of the nature of your defects or even productivity. So if you're not in a scenario where you do have a lot of teams, you do have a lot of like developers pushing codes, code changes all the time. Uh, maybe your bottleneck, maybe your concerns are actually on the development side. So you should be looking at metrics on that side, like code quality, or product quality in general, defect density, uh, productivity, these sorts of things.
Kovid Batra: I think great point there. Uh, actually, context is what is most important and DORA could be the first step to look into engineering efficiency in general, but the important, or I should say the real point is understanding the context and then applying the metrics and we would need metrics which are on DORA also. Like, as you mentioned, like there would be scenarios where you would want to look at defect density, you would want to look at code quality, and from that, uh, I think one of the interesting, uh, metrics that I have recently come across is about code collaboration also, right? So people look at how well the teams are collaborating over the code reviews. So that also becomes an essential part of when you're shipping your software, right? So the quality gets impacted. The velocity of the delivery gets impacted. Have you encountered a scenario where you wanted or you had measured code review collaboration within the team? And if you did so, uh, how did you do it?
Peter Peret Lupo: Yes, actually in different ways. So one thing that I like to do, it's more of a qualitative measurement, but I do believe there is space for this kind of metric as well. One thing that I like doing, that I'm currently doing, and I've done in other companies as well, is taking some part of the Sprint's retrospective to share with the team, results of a survey. And one of the things that I do ask on the survey is if they're being supported by team members, if they're supporting team members. So it's just like a Likert Scale, like 1 to 5, but it highlights like that kind of collaboration support.
Kovid Batra: Right.
Peter Peret Lupo: Um, it's anonymous, so I can't tell like who is helping who. Uh, so sometimes somebody's, like, being very, like being helped a lot, and sometimes some other person is helping a lot. And maybe they switch, depending on like whether or not they're working on something that they're familiar with and the other person isn't or vice versa, I don't know. I have no means to do that, and I don't bother about that. Nobody should be bothering about that. I think if you have like a very senior person, they're probably like helping a lot of people and maybe they're not pushing many changes, but like everybody relies on them. Uh, so if you're working on the same, you should be measuring the team, right? But there are other things as well, like, um, you can see like comments on code reviews, who jumps to do code reviews, and all those kinds of things, right? These are very important indicators that they have like a healthy team, that they're supporting each other. You can even like indicate some things like if people are getting, uh, are learning more about the codes component they are changing or like some, like a service or whatever area, how you define it, uh, if you have like knowledge silos and, um, who should be providing training to whom to break out those silos to improve productivity. So yeah, that's very insightful and very helpful. Yeah, definitely.
Kovid Batra: Makes sense, makes sense Um, is there anything that you have used, uh, to look at the technical debt? So that is also something I have, uh, always heard from managers and leaders. Like when you're building, whether you are a large organization or you are a small one moving faster, uh, the degree might vary, but you accumulate technical debt over a period of time. Is there something that you think could be looked at as a metric to indicate that, okay, it's high time now, that we should look at technical debt? Because mostly what happens is like whenever there are team meetings, people just come up with ideas that, okay, this is what we can improve, this is where we are facing a lot of bugs and issues. So let's work on this piece because this has now become a debt for us, but is there something objective that could tell that yes, now it's time that we should sit down and look at the technical debt part?
Peter Peret Lupo: Well, uh, the problem is like, there are so many, uh, different approaches to technical debt. They're going to be more suited to one organization or another organization. If you have like a very, uh, engineering-driven organization, you tend to have less technical debt or you tend to pay that technical debt more often. But if it's not the case, if it's like more product-driven, you tend to accumulate those more often, and then you need to apply different approaches. So, one thing that I like doing is like when we are acquiring the debt; and that's normal, that's part of life. Sometimes you have to, and you should be okay with that. But when we are acquiring debt, we catalog it somewhere. Maybe you have like an internal wiki or something, like whatever documentation tool you use. You add that to a catalog where you basically have like your components or services or however you split your application. And then like what's the technical data you're acquiring, which would be the appropriate solutions or alternatives, how that's going to impact, and most importantly, when you believe you should pay that so you don't get like a huge impact, right?
Kovid Batra: Right. Of course. So just one thing I recently heard from one of my friends. Like they look at the time for a new developer to do the first commit as an indicator of technical debt. So if they.. First release, actually. So if someone who is joining new in the team, if they're taking too much time to reach a point where they could actually merge their code, and like have it on production, uh, if that is high and they, of course, keep a baseline there, then they consider that there is a lot of debt they might have accumulated because of which the learning and the implementation for the first release from a new developer is taking time. So do you think this approach could work or this approach could be inferential to what we are talking about, like the technical debt?
Peter Peret Lupo: I think that in this particular case, there are so many confounding variables. People join the team at different seniority levels. A senior might take less time than a junior, even in a scenario where there is more technical debt. So that alone is hard to compare. Even at the same level, people join with different skills. So maybe you have like a feature you need to write frontend and backend code, and some people are, uh, full stack but are more backend-inclined, more frontend-inclined. That alone will change your metric. You are waiting for one person to join that team so you can have like a new point of measurement. So you're not gonna have a lot, and there's gonna have like a lot of variation because of these confounding factors. Even that the onboarding process may change in between. The way that I usually like to introduce people to code is asking them to reduce the amount of warnings from like code linters first, and then fixing some simple defects, and then something like a more complex defect, and then writing a new feature. Uh, so, even like depending on your own onboarding strategy, your model strategy you're defining is going to affect that metric. So I wouldn't be very confident on that metric for this purpose.
Kovid Batra: Okay. Got it, got it. Makes sense. All right. I think if I have to ask you, uh, it's never easy, right? Like in the beginning, you mentioned that the first point itself is talking about these metrics is hard, right? Even if they make a lot of practical sense, but talking about it is hard. So when there is inherited resistance towards this topic, uh in the teams, when you go about implementing it, there could be a hell of a lot of challenges, right? And I'm sure, you would have also come across some of those in your journey when you were implementing it. So can you give us some examples from the implementation point of view, like how does the implementation go for, uh, these metrics and what are the challenges one faces when they're implementing it? And maybe if there are certain timelines to which one should look at for a full-fledged implementation and getting some success from the implementation of these metrics.
Peter Peret Lupo: Right. So, um, usually you're measuring something because you want to prove something, right? Because you want to like achieve like a certain goal, uh, maybe organizational, or just like the team. So I think that the first thing to lower, uh, the resistance is having a clear goal, and making sure that everybody understands that, uh, that the goal is not measuring anybody, uh, individually. That already like reduces the resistance a lot, and making sure that people understand why that goal is important and how you're going to measure in it is also extremely important.
Another thing that is interesting is to ask people for inputs on like how they think you could be measuring that. So making them also part of the process, and maybe the way that they're advising is not going to be like the way that you end up measuring. Maybe it influences, maybe it's exactly what they suggest, but the important thing is to make them part of the process, so they don't feel that the process, like the process of establishing metrics is not something that is being done to them, but something that they are doing with everybody else.
And so honestly, like so many things are already measured by the team, uh, velocity or however they estimate productivity. Even the estimates themselves are on like tickets on user stories or, uh, these are all, uh, attempts to measure things and they're used to compare the destinations with, uh, the actual results, so they know what the measures are used for. So sometimes it's just a matter of like establishing these parallels. Look, we measure productivity, we measure velocity to see if we are getting better, if we're getting worse. We also need to measure, uh, the quality to see if we're like catching more defects than before, if we have like more escape defects. Measurement is in some way already a part of our lives. Most of the times, it's a matter of like highlighting that, and, uh, people are usually comfortable with them, yeah, once you go through all this.
Kovid Batra: Right. Makes sense. Um, I think the major part is done when the team is aligned on the 'why' part, like why you are doing it, because as soon as they realize that there is some importance to measuring this metric, they would automatically be, uh, intuitively be aligned towards measuring that, and it becomes easier because then if there are challenges related to the implementation process also, they would like come together and maybe find out ways to, uh, build things around that and help in actual measurement of the metric also.
But if I have to ask, let's say a team is fully aligned and, uh, we are looking at implementing, let's say DORA metrics for a team, what should be the time frame one should keep in mind to get an understanding of what these metrics are saying? Because it's not like straightforward. You look at the common frequency, if it's high, you say things are good. If it's low, things are bad. Of course, it doesn't work like that. You have to understand these metrics in the first place in the cadence of your team, in the situation of your team, and then make sense out of it and find out those bottlenecks or those areas of inefficiency where you could really work upon, right? So what should be that time frame in one's mind that someone is an engineering manager who is implementing this for a team? What time frame should that person keep in mind and what exactly should be the first step towards measuring these once you start implementing them?
Peter Peret Lupo: Right. So it's a very good question. Time frame varies a lot and I'll tell you why; because more important than the time is the amount of data points that you have. If you wait for, like let's say, a month and you have like three data points, you can't establish any sort of pattern. You don't know if that's increasing, decreasing. There's no confidence. There's no statistical relevance. It's, like, the sample is too small. But like if you collect, like three data points, that's like generic for any metric. If you collect, like three data points every day, maybe in a week you'll have enough. The problem I see here is like, let's say, uh, something happens that is out of the ordinary. I want to investigate that to see if there is room for improvement there, or if that actually indicates that something went like really well and you want to replicate that success in the other cases. Um, you can't tell what's out of the ordinary if you're looking at three, four points.
Kovid Batra: Right.
Peter Peret Lupo: Uh, or if it's just like normal variation. So, I think that what's important is to have like a good baseline. So, that's gonna vary from process to process, from organization to organization, but there are some indications in the literature that like you should collect at least 30 data points. I think that with 30 data points you have like somewhat of a good, uh, statistical relevance for it, for your analysis, but I don't think you should, you have to wait for those many points in order to start investigating things. Sometimes you have like 10 or 12 and you already see something that. looks like something that you should investigate or you start having like an idea of what's going on, if it's higher than you expected, if it's lower than you expected, and you can start taking actions and investigating that as long as you consider that your interpretation may not be valid, bec ause like your sample is small. The time that it takes, like time frame, I guess that's going to depend on how often you are able to collect a new data point, and that's going to vary from organization to organization and from process to process, like measuring quality is different from measuring productivity, uh, and so on. So, I think all these things need to be taken into consideration. I think that the confidence is important.
And one other thing that you mentioned there, about like the team analyzing. It's something that I want to touch on because it's an experience that I've had more than once. You mentioned context. Context is super important. So much so that I think that the team that is producing the metrics should be the first one looking at them, not management, higher management, C-level, not them, because they are the only ones that are able to look at data points and say, "Yeah, things here didn't go well. Our only QA was on vacation." Or like somebody took a sick day or whatever reason, like they have the context. So they should be the first ones looking at the metric, analyzing the metric, and conveying the results of their analysis to higher levels, not the other way around, because what happens when you have it the other way around is that, like, they don't have the context, so they're looking at just the numbers, and if the number is bad, they're gonna inquire about it. If it's good, they are usually gonna stay quiet, uh, and they're gonna ask about the bad numbers, whether or not there was a good reason for that, whether or not it was like, uh, let's say, an exception. And then the team is going to feel that they have to defend themselves, to justify themselves every time, and it creates like a very poisonous scenario where the team feels that management is there to question them and they need to defend themselves against management instead of them having the autonomy to report on their success and their failures to management and let management deal with those results instead of the causes.
Kovid Batra: Totally, totally.
Peter Peret Lupo: Context is super important.
Kovid Batra: Great point there. Yeah, of course. Great point there, uh, highlighting the do's and don'ts from your experience and it's very relevant actually because the numbers don't always give you the reality of the situation. They could be an indicator and that's why we have them in place. Like first thing, you measure it. Don't come to a conclusion from it directly. If you see some discrepancy, like if there are some extraordinary data points, as you said, then there is a point which you should come out and inquire to understand what exactly happened here, but not directly jump onto the team saying that, Oh, you're not doing good or the other way around. So I think that that totally makes sense, uh, Peter.
I think it was really, really interesting talking to you about the metrics and the implementation and the experiences that you have shared. Um, we could go on on this, but today I think we'll have to stop here and, uh, say goodbye to you. Maybe we can have another round of discussion continuing with those experiences that you have had with the implementation.
Peter Peret Lupo: Definitely. It was a real pleasure..
Kovid Batra: It would be our pleasure, actually. But, uh, like before you leave, uh, anything that you want to share with our audience as parting advice, uh, would be really appreciated.
Peter Peret Lupo: All right. Um, look at your metrics as an ally, as a guide to tell you where you're going. Compare what you're doing now with what you were doing before to see if you're improving. When I say 'you', I'm talking to, uh, each individual in the team. Consider your team metrics, look at them, your work is part of the work that is being analyzed, and you have an influence on that at an individual level and with your team. So do look at your metrics, compare where you are at with where you were before to see if your changes were improved, see if your changes, uh, carried improvements you're looking for, and talk to your team about these metrics on your sprint retrospective. That's a very powerful tool to tell you, like, if your, uh, retrospective actions are being effective in delivering the change that you want in your process.
Kovid Batra: Great! I think great piece of advice there. Thanks, Peter. Thank you so much. Uh, this was really insightful. Loved talking to you.
Peter Peret Lupo: All right. Thank you.
Implement DORA metrics, gain dev visibility & drive success in your teams.
Try Live Demo
