In this episode of the groCTO Podcast, host Kovid Batra engages in a comprehensive discussion with Geoffrey Teale, the Principal Product Engineer at Upvest, who brings over 25 years of engineering and leadership experience.
The episode begins with Geoffrey's role at Upvest, where he has transitioned from Head of Developer Experience to Principal Product Engineer, emphasizing a holistic approach to improving both developer experience and engineering standards across the organization. Upvest's business model as a financial infrastructure company providing investment banking services through APIs is also examined. Geoffrey underscores the multifaceted engineering requirements, including security, performance, and reliability, essential for meeting regulatory standards and customer expectations. The discussion further delves into the significance of product thinking for internal teams, highlighting the challenges and strategies of building platforms that resonate with developers' needs while competing with external solutions.
Throughout the episode, Geoffrey offers valuable insights into the decision-making processes, the importance of simplicity in early-phase startups, and the crucial role of documentation in fostering team cohesion and efficient communication. Geoffrey also shares his personal interests outside work, including his passion for music, open-source projects, and low-carbon footprint computing, providing a holistic view of his professional and personal journey.
Kovid Batra: Hi, everyone. This is Kovid, back with another episode of groCTO Podcast. Today with us, we have a very special guest who has great expertise in managing developer experience at small scale and large scale organizations. He is currently the Principal Engineer at Upvestm, and has almost 25 plus years of experience in engineering and leadership. Welcome to the show, Geoffrey. Great to have you here.
Geoffrey Teale: Great to be here. Thank you.
Kovid Batra: So Geoffrey, I think, uh, today's theme is more around improving the developer experience, bringing the product thinking while building the platform teams, the platform. Uh, and you, you have been, uh, doing all this from quite some time now, like at Upvest and previous organizations that you've worked with, but at your current company, uh, like Upvest, first of all, we would like to know what kind of a business you're into, what does Upvest do, and let's then deep dive into how engineering is, uh, getting streamlined there according to the business.
Geoffrey Teale: Yeah. So, um, Upvest is a financial infrastructure company. Um, we provide, uh, essentially investment banking services, a complete, uh, solution for building investment banking experiences, uh, for, for client organizations. So we're business to business to customer. We provide our services via an API and client organizations, uh, names that you'd heard of people like Revolut and N26 build their client-facing applications using our backend services to provide that complete investment experience, um, currently within the European Union. Um, but, uh, we'll be expanding out from there shortly.
Kovid Batra: Great. Great. So I think, uh, when you talk about investment banking and supporting the companies with APIs, what kind of engineering is required here? Is it like more, uh, secure-oriented, secure-focused, or is it more like delivering on time? Or is it more like, uh, making things very very robust? How do you see it right now in your organization?
Geoffrey Teale: Well, yeah, I mean, I think in the space that we're in the, the answer unfortunately is all of the above, right? So all those things are our requirements. It has to be secure. It has to meet the, uh, the regulatory standards that we, we have in our industry. Um, it has to be performant enough for our customers who are scaling out to quite large scales, quite large numbers of customers. Um, has to be reliable. Um, so there's a lot of uh, uh, how would I say that? Pressure, uh, to perform well and to make sure that things are done to the highest possible standard in order to deliver for our customers. And, uh, if we don't do that, then, then, well, the customers won't trust us. If they don't trust us, then we wouldn't be where we are today. So, uh, yeah.
Kovid Batra: No, I totally get that. Uh, so talking more about you now, like, what's your current role in the organization? And even before that, tell us something about yourself which the LinkedIn doesn't know. Uh, I think the audience would love to know you a little bit more. Uh, let's start from there. Uh, maybe things that you do to unwind or your hobbies or you're passionate about anything else apart from your job that you're doing?
Geoffrey Teale: Oh, well, um, so, I'm, I'm quite old now. I have a family. I have two daughters, a dog, a cat, fish, quail. Keep quail in the garden. Uh, and that occupies most of my time outside of work. Actually my passions outside of work were always um, music. So I play guitar, and actually technology itself. So outside of work, I'm involved and have been involved in, in open source and free software for, for longer than I've been working. And, uh, I have a particular interest in, in low carbon footprint computing that I pursue outside of, out of work.
Kovid Batra: That's really amazing. So, um, like when you say low carbon, uh, cloud computing, what exactly are you doing to do that?
Geoffrey Teale: Oh, not specifically cloud computing, but that would be involved. So yeah, there's, there's multiple streams to this. So one thing is about using, um, low power platforms, things like RISC-V. Um, the other is about streamlining of software to make it more efficient so we can look into lots of different, uh, topics there about operating systems, tools, programming languages, how they, uh, how they perform. Um, sort of reversing a trend, uh, that's been going on for as long as I've been in computing, which is that we use more and more power, both in terms of computing resource, but also actual electricity for the network, um, to deliver more and more functionality, but we're also programming more and more abstracted ways with more and more layers, which means that we're actually sort of getting less, uh, less bang for buck, if you, if you like, than we used to. So, uh, trying to reverse those trends a little bit.
Kovid Batra: Perfect. Perfect. All right. That's really interesting. Thanks for that quick, uh, cute little intro. Uh, and, uh, now moving on to your work, like we were talking about your experience and your specialization in DevEx, right, improving the developer experience in teams. So what's your current, uh, role, responsibility that comes with, uh, within Upvest? Uh, and what are those interesting initiatives that you have, you're working on?
Geoffrey Teale: Yeah. So I've actually just changed roles at Upvest. I've been at Upvest for a little bit over two years now, and the first two years I spent as the Head of Developer Experience. So running a tribe with a specific responsibility for client-facing developer experience. Um, now I've switched into a Principal Engineering role, which means that I have, um, a scope now which is across the whole of our engineering department, uh, with a, yeah, a view for improving experience and improving standards and quality of engineering internally as well. So, um, a slight shift in role, but my, my previous five years before, uh, Upvest, were all in, uh, internal development experience. So I think, um, quite a lot of that skill, um, coming into play in the new role which um, yeah, in terms of challenges actually, we're just at the very beginning of what we're doing on that side. So, um, early challenges are actually about identifying what problems do exist inside the company and where we can improve and how we can make ourselves ready for the next phase of the company's lifetime. So, um, I think some of those topics would be quite familiar to any company that's relatively modern in terms of its developer practices. If you're using microservices, um, there's this aspect of Conway's law, which is to say that your organizational structure starts to follow the program structure and vice versa. And, um, in that sense, you can easily get into this world where teams have autonomy, which is wonderful, but they can be, um, sort of pushed into working in a, in a siloized fashion, which can be very efficient within the team, but then you have to worry about cohesion within the organization and about making sure that people are doing the right things, uh, to, to make the services work together, in terms of design, in terms of the technology that we develop there. So that bridges a lot into this world of developer experience, into platform drives, I think you mentioned already, and about the way in which you think about your internal development, uh, as opposed to just what you do for customers.
Kovid Batra: I agree. I mean, uh, as you said, like when the teams are siloed, they might be thinking they are efficient within themselves. And that's mostly the use case, the case. But when it comes to integrating different pieces together, that cohesion has to fall in. What is the biggest challenge you have seen, uh, in, in the teams in the last few years of your experience that prevents this cohesion? And what is it that works the best to bring in this cohesion in the teams?
Geoffrey Teale: Yeah. So I think there's, there's, there's a lot of factors there. The, the, the, the biggest one I think is pressure, right? So teams in most companies have customers that they're working for, they have pressure to get things done, and that tends to make you focus on the problem in front of you, rather than the bigger picture, right? So, um, dealing, dealing with that and reinforcing the message to engineers that it's actually okay to do good engineering and to worry about the other people, um, is a big part of that. I've always said, actually, that in developer experience, a big part of what you have to do, the first thing you have to do is actually teach people about why developer experience is important. And, uh, one of those reasons is actually sort of saying, you know, promoting good behavior within engineering teams themselves and saying, we only succeed together. We only do that when we make the situation for ourselves that allows us to engineer well. And when we sort of step away from good practice and rush, rush, um, that maybe works for a short period of time. But, uh, in the long term that actually creates a situation where there's a lot of mess and you have to deal with, uh, getting past, we talk about factors like technical debt. There's a lot of things that you have to get past before you can actually get on and do the productive things that you want to do. Um, so teaching organizations and engineers to think that way is, uh, is, uh, I think a big, uh, a big part of the work that has to be done, finding ways to then take that message and put it into a package that is acceptable to people outside of engineering so that they understand why this is a priority and why it should be worked on is, I think, probably the second biggest part of that as well.
Kovid Batra: Makes sense. I think, uh, most of the, so is it like a behavioral challenge, uh, where, uh, developers and team members really don't like the fact that they have to work in cohesion with the teams? Or is it more like the organizational structure that put people into a certain kind of mindset and then they start growing with that and that becomes a problem in the later phase of the organization? What, what you have seen, uh, from your experience?
Geoffrey Teale: Yeah. So I mean, I think growth is a big part of this. So, um, I mean, I've, I've worked with a number of startups. I've also worked in much bigger organizations. And what happens in that transition is that you move from a small tight-knit group of people who sort of inherently have this very good interpersonal communication, they all know what's going on with the company as a whole, and they build trust between them. And that way, this, this early stage organization works very well, and even though you might be working on disparate tasks, you always have some kind of cohesion there. You know what to do. And if something comes up that affects all of you, it's very easy to identify the people that you need to talk to and find a solution for it. Then as you grow, you start to have this situation where you start to take domains and say, okay, this particular part of, of what we do now belongs in a team, it has a leader and this piece over here goes over there. And that still works quite well up into a certain scale, right? But after time in an organization, several things happen. Okay, so your priorities drift apart, right? You no longer have such good understanding of the common goal. You tend to start prioritizing your work within those departments. So you can have some, some tension between those goals. It's not always clear that Department A should be working together with Department B on the same priority. You also have natural staff turnover. So those people who are there at the beginning, they start to leave, some of them, at least, and these trust relationships break down, the communication channels break down. And the third factor is that new people coming into the organization, they haven't got these relationships, they haven't got this experience. They usually don't have, uh, the position to, to have influence over things on such a large scale. So they get an expectation of these people that they're going to be effective across the organization in the way that people who've been there a long time are, and it tends not to happen. And if you haven't set up for that, if you haven't built the support systems for that and the internal processes and tooling for that, then that communication stops happening in the way that it was happening before.
So all of those things create pressure to, to siloes, then you put it on the pressure of growth and customers and, and it just, um, uh, ossifies in that state.
Kovid Batra: Totally. Totally. And I think, um, talking about the customers, uh, last time when we were discussing, uh, you very beautifully put across this point of bringing that product thinking, not just for the products that you're building for the customer, but when you're building it for the teams. And I, what I feel is that, the people who are working on the platform teams have come across this situation more than anyone else in the team as a developer, where they have to put in that thought of product thinking for the people within the team. So what, what, what, uh, from where does this philosophy come? How you have fitted it into, uh, how platform teams should be built? Just tell us something about that.
Geoffrey Teale: Yeah. So this is something I talk about a little bit when I do presentations, uh, about developer experience. And one of the points that I make actually, particularly for platform teams, but any kind of internal team that's serving other internal teams is that you have to think about yourself, not as a mandatory piece that the company will always support and say, "You must use this, this platform that we have." Because I have direct experience, not in my current company, but in previous, uh, in previous employers where a lot of investment has been made into making a platform, but no thought really was given to this kind of developer experience, or actually even the idea of selling the platform internally, right? It was just an assumption that people would have to use it and so they would use it. And that creates a different set of forces than you'll find elsewhere. And, and people start to ignore the fact that, you know, if you've got a cloud platform in this case, um, there is competition, right? Every day as an engineer, you run into people out there working in the wide world, working for, for companies, the Amazons, AWS of this world, as your Google, they're all producing cloud platform tools. They're all promoting their cloud native development environments with their own reasons for doing that. But they expend a lot of money developing those things, developing them to a very high standard and a lot of money promoting and marketing those things. And it doesn't take very much when we talk just now about trust breaking down, the cohesion between teams breaking down. It doesn't take very much for a platform to start looking like less of a solution and more of a problem if it's taking you a long time to get things done, if you can't find out how to do things, if you, um, you have bad experiences with deployment. This all turns that product into an internal problem.
Kovid Batra: In context of an internal problem for the teams.
Geoffrey Teale: Yeah, and in that context, and this is what I, what I've seen, when you then either have someone coming in from outside with experience with another, a product that you could use, or you get this kind of marketing push and sales push from one of these big companies saying, "Hey, look at this, this platform that we've got that you could just buy into." um, it, it puts you in direct competition and you can lose that, that, right? So I have seen whole divisions of a, of a very large company switch away from the internal platform to using cloud native development, right, on, on a particular platform. Now there are downsides for that. There are all sorts of things that they didn't realize they would have to do that they end up having to do. But once they've made the decision, that battle is lost. And I think that's a really key topic to understand that you are in competition, even though you're an internal team, you are in competition with other people, and you have to do some of the things that they do to convince the people in your organization that what you're doing is beneficial, that it's, it's, it's useful, and it's better in some very distinct way than what they would get off the shelf from, from somewhere else.
Kovid Batra: Got it. Got it. So, when, uh, whenever the teams are making this decision, let's, let's take something, build a platform, what are those nitty gritties that one should be taking care of? Like, either people can go with off the shelf solutions, right? And then they start building. What, what should be the mindset, what should be the decision-making mindset, I must say, uh, for, for this kind of a process when they have to go through?
Geoffrey Teale: So I think, um, uh, we within Upvest, follow a very, um, uh, prescribed is not the right word, but we have a, we have a process for how we think about things, and I think that's actually a very useful example of how to think about any technical project, right? So we start with this 'why' question and the 'why' question is really important. We talk about product thinking. Um, this is, you know, who are we doing this for and what are the business outcomes that we want to achieve? And that's where we have to start from, right? So we define that very, very clearly because, and this is a really important part, there's no value, uh, in anybody within the organization saying, "Let's go and build a platform." For example, if that doesn't deliver what the company needs. So you have to have clarity about this. What is the best way to build this? I mean, nobody builds a platform, well not nobody, but very few people build a platform in the cloud starting from scratch. Most people are taking some existing solution, be that a cloud native solution from a big public cloud, or be that Kubernetes or Cloud Foundry. People take these tools and they wrap them up in their own processes, their own software tools around it to package them up as a, uh, a nice application platform for, for development to happen, right? So why do you do that? What, what purpose are you, are you serving in doing this? How will this bring your business forward? And if you can't answer those questions, then you probably should never even start the project, right? That's, that's my, my view. And if you can't continuously keep those, um, ideas in mind and repeat them back, right? Repeat them back in terms of what are we delivering? What do we measure up against to the, to the, to the company? Then again, you're not doing a very good job of, of, of communicating why that product exists. If you can't think of a reason why your platform delivers more to your company and the people working in your company than one of the off the shelf solutions, then what are you for, right? That's the fundamental question.
So we start there, we think about those things well before we even start talking about solution space and, and, um, you know, what kind of technology we're going to use, how we're going to build that. That's the first lesson.
Kovid Batra: Makes sense. A follow-up question on that. Uh, let's say a team is let's say 20-30 folks right now, okay? I'm talking about an engineering team, uh, who are not like super-funded right now or not in a very profit making business. This comes with a cost, right? You will have to deploy resources. You will have to invest time and effort, right? So is it a good idea according to you to have shared resources for such an initiative or it doesn't work out that way? You need to have dedicated resources, uh, working on this project separately or how, how do you contemplate that?
Geoffrey Teale: My experience of early-phase startups is that people have to be multitaskers and they have to work on multiple things to make it work, right? It just doesn't make sense in the early phase of a company to invest so heavily in a single solution. Um, and I think one of the mistakes that I see people making now actually is that they start off with this, this predefined idea of where they're going to be in five years. And so they sort of go away and say, "Okay, well, I want my, my, my system to run on microservices on Kubernetes." And they invest in setting up Kubernetes, right, which has got a lot easier over the last few years, I have to say. Um, you can, to some degree, go and just pick that stuff off the shelf and pay for it. Um, but it's an example of, of a technical decision that, that's putting the cart before the horse, right? So, of course, you want to make architectural decisions. You don't want to make investments on something that isn't going to last, but you also have to remember that you don't know what's going to happen. And actually, getting to a product quickly, uh, is more important than, than, you know, doing everything perfectly the first time around. So, when I talk about these, these things, I think uh, we have to accept that there is a difference between being like the scrappy little startup and then being in growth phase and being a, a mega corporation. These are different environments with different pressures
Kovid Batra: Got it. So, when, when teams start, let's say, work on it, working on it and uh, they have started and taken up this project for let's say, next six months to at least go out with the first phase of it. Uh, what are those challenges which, uh, the platform heads or the people who are working, the engineers who are working on it, should be aware of and how to like dodge those? Something from your experience that you can share.
Geoffrey Teale: Yes. So I mean, in, in, in the, the very earliest phase, I mean, as I just alluded to that keeping it simple is, is a, a, a big benefit. And actually keeping it simple sometimes means, uh, spending money upfront. So what I've, what I've seen is, is, um, many times I've, I've worked at companies, um, but so many, at least three times who've invested in a monitoring platform. So they've bought a off the shelf software as a service monitoring platform, uh, and used that effectively up until a certain point of growth. Now the reason they only use it up into a certain point of growth is because these tools are extremely expensive and those costs tend to scale with your company and your organization. And so, there comes a point in the life of that organization where that no longer makes sense financially. And then you withdraw from that and actually invest in, in specialist resources, either internally or using open source tools or whatever it is. It could just be optimization of the tool that you're using to reduce those costs. But all of those things have a, a time and financial costs associated with them. Whereas at the beginning, when the costs are quite low to use these services, it actually tends to make more sense to just focus on your own project and, and, you know, pick those things up off the shelf because that's easier and quicker. And I think, uh, again, I've seen some companies fail because they tried to do everything themselves from scratch and that, that doesn't work in the beginning. So yeah, I think that's a, it's a big one.
The second one is actually slightly later as you start to grow, getting something up and running at all is a challenge. Um, what tends to happen as you get a little bit bigger is this effect that I was talking about before where people get siloized, um, the communication starts to break down and people aren't aware of the differing concerns. So if you start worrying about things that you might not worry about at first, like system recovery, uh, compliance in some cases, like there's laws around what you do in terms of your platform and your recoverability and data protection and all these things, all of these topics tend to take focus away, um, from what the developers are doing. So on the first hand, that tends to slow down delivery of, of, features that the engineers within your company want in favor of things that they don't really want to know about. Now, all the time you're doing this, you're taking problems away from them and solving them for them. But if you don't talk about that, then you're not, you're not, you may be delivering value, but nobody knows you're delivering value. So that's the first thing.
The other thing is that you then tend to start losing focus on, on the impact that some of these things have. If you stop thinking about the developers as the primary stakeholders and you get obsessed about these other technical and legal factors, um, then you can start putting barriers into place. You can start, um, making the interfaces to the system the way in which it's used, become more complicated. And if you don't really focus then on the developer experience, right, what it is like to use that platform, then you start to turn into the problem, which I mentioned before, because, um, if you're regularly doing something, if you're deploying or testing on a platform and you have to do that over and over again, and it's slowed down by some bureaucracy or some practice or just literally running slowly, um, then that starts to be the thing that irritates you. It starts to be the thing that's in your way, stopping you doing what you're doing. And so, I mean, one thing is, is, is recognizing when this point happens, when your concerns start to deviate and actually explicitly saying, "Okay, yes, we're going to focus on all these things we have to focus on technically, but we're going to make sure that we reserve some technical resource for monitoring our performance and the way in which our customers interact with the system, failure cases, complaints that come up often."
Um, so one thing, again, I saw in much bigger companies, is they migrated to the cloud from, from legacy systems in data centers. And they were used to having turnaround times on, on procedures for deploying software that took at least weeks or having month-long projects because they had to wait for specific training that they had to get sign off. And they thought that by moving to an internal cloud platform, they would solve these things and have this kind of rapid development and deployment cycle. They sort of did in some ways, but they forgot, right? When they were speculating out, they forgot to make the developers a stakeholder and saying, "What do you need to achieve that?" And what they actually need to achieve that is a change in the mindset around the bureaucracy that came around. It's all well and good, like not having to physically put a machine in a rack and order it from a company. But if you still have these rules that say, okay, you need to go in this training course before you can do anything with this, and there's a six month waiting list for that training course, or this has to be approved by five managers who can only be contacted by email before you can do it. These processes are slowing things down. So actually, I mentioned that company that, uh, we lost the whole department from the, from the, uh, platform that we had internally. One of the reasons actually was that just getting started with this platform took months. Whereas if you went to a public cloud service, all you needed was a credit card and you could do it and you wouldn't be breaking any rules in the company in doing that. As long as you had the, the right to spend the money on the credit card, it was fine.
So, you know, that difference of experience, that difference of, uh, of understanding something that starts to grow out as you, as you grow, right? So I think that's a, uh, a thing to look out for as you move from the situation when you're 10, 20 people in the whole company to when you're about, I would say, 100 to 200 people in the whole company. These forces start to become apparent.
Kovid Batra: Got it. So when, when you touch that point of 100-200, uh, then there is definitely a different journey that you have to look up to, right? And there are their own set of challenges. So from that zero to one and then one to X, uh, journey, what, what things have you experienced? Like, this would be my last question for, for today, but yeah, I would be really interested for people who are listening to you heading teams of sizes, a hundred and above. What kind of things they should be looking at when they are, let's say, moving from an off the shelf to an in-house product and then building these teams together?
Geoffrey Teale: Oh, what should they be looking at? I mean, I think we just covered, uh, one of the big ones. I'd say actually that one of the, the biggest things for engineers particularly, um, and managers of engineers is resistance to documentation and, and sort of ideas about documentation that people have. So, um, when you're again, when you're that very small company, it's very easy to just know what's going on. As you grow, what happens, new people come into your team and they have the same questions that have been asked and answered before, or were just known things. So you get this pattern where you repeatedly get the same information being requested by people and it's very nice and normal to have conversations. It builds teams. Um, but there's this kind of key phrase, which is, 'Documentation is automation', right? So engineers understand automation. They understand why automation is required to scale, but they tend to completely discount that when it comes to documentation. So almost every engineer that I've ever met hates writing documentation. Not everyone, but almost everyone. Uh, but if you go and speak to engineers about what they need to start working with a new product, and again, we think about this as a product, um, they'll say, of course, I need some documentation. Uh, and if you dive into that, they don't really want to have fancy YouTube videos. And so, that sometimes that helps people overcome a resistance to learning. Um, but, uh, having anything at all is useful, right? But this is a key, key learning documentation. You need to treat it a little bit like you treat code, right? So it's a very natural, um, observation from, from most engineers. Well, if I write a document about this, that document is just going to sit there and, and rot, and then it will be worse than useless because it will say the wrong thing, which is absolutely true. But the problem there is that someone said it will sit there and rot, right? It shouldn't be the case, right? If you need the documentation to scale out, you need these pieces to, to support new people coming into the company and to actually reduce the overhead of communication because more people, the more different directions of communication you have, the more costly it gets for the organization. Documentation is boring. It's old-fashioned, but it is the solution that works for fixing that.
The only other thing I'm going to say about is mindset, is it's really important to teach engineers what to document, right? Get them away from this mindset that documentation means writing massive, uh, uh, reams and reams of, of text explaining things in, in detail. It's about, you know, documenting the right things in the right place. So at code-level, commenting, um, saying not what the code there does, but more importantly, generally, why it does that. You know, what decision was made that led to that? What customer requirement led to that? What piece of regulation led to that? Linking out to the resources that explain that. And then at slightly higher levels, making things discoverable. So we talk actually in DevEx about things like, um, service catalogs so people can find out what services are running, what APIs are available internally. But also actually documentation has to be structured in a way that meets the use cases. And so, actually not having individual departments dropping little bits of information all over a wiki with an arcane structure, but actually sort of having a centralized resource. Again, that's one thing that I did actually in a bigger company. I came into the platform team and said, "Nobody can find any information about your platform. You actually need like a central website and you need to promote that website and tell people, 'Hey, this is here. This is how you get the information that you need to understand this platform.' And actually including at the very front of that page why this platform is better than just going out somewhere else to come back to the same topic."
Documentation isn't a silver bullet, but it's the closest thing I'm aware of in tech organizations, and it's the thing that we routinely get wrong.
Kovid Batra: Great. I think, uh, just in the interest of time, we'll have to stop here. But, uh, Geoffrey, this was something really, really interesting. I also explored a few things, uh, which were very new to me from the platform perspective. Uh, we would love to, uh, have you for another episode discussing and deep diving more into such topics. But for today, I think this is our time. And, uh, thank you once again for joining in, taking out time for this. Appreciate it.
Geoffrey Teale: Thank you. It's my pleasure.
Implement engineering metrics, gather SDLC insights & set continuous improvement goals in your dev teams.
Try Live Demo
