Scaling DevOps in the Enterprise: An Interview with Anders Wallgren

Josiah Renaudin: Welcome back to another TechWell interview. Today I'm joined by Anders Wallgren, the CTO of Electric Cloud. We'll be speaking on scaling DevOps in the enterprise.

Anders, thank you very much for joining me today.

Anders Wallgren: My pleasure.

Josiah Renaudin: First, before we kind of get really into the context, tell us a bit about your experience in the industry.

Anders Wallgren: I've been working in the software industry for "mumble mumble years now," they say. I've worked all up and down the stack over the course of my years: UI, network stacks, native applications, web applications, back end, you name it. I think one of the interesting things is that all the companies that I've worked at, we always somehow seem to be a little bit on the bleeding edge of how to do our processes better, how do we get out of our own way and produce more software for our customers.

It's been really interesting the last few years, working with Electric Cloud and with our customers and the community in general, just to see how vastly the processes and the culture and the technology has changed over the years. It's really kind of cool to be part of that.

Josiah Renaudin: One of those changes, one of those things that I feel like I've written a lot about at this point, I keep reading a lot about, is DevOps. That's kind of the focus of this conversation today, so to kick things off: Why has DevOps started as such a development-driven movement, since ops—you know, operations—is right in the name?

Anders Wallgren: No, that's true, and I think ultimately, to succeed, it has to be something that covers both. I think part of the reason why a lot of people, myself included, kind of look at it as a ... at least initiated in dev, to a large degree, is simply because that's where agile took root first.

As organizations started to adopt agile and get better at delivering the software, at least out of the development and QA organizations, the logical next question was, "Well, how do we get it into customers' hands more quickly?" Whether that's up on a website, or burned into a chip that goes into a box that you buy on Amazon, or what have you.

I think it's a little bit natural that it started there and if you speak to pure agilists they'll happily inform you that the Agile Manifesto always talked about delivering software to customers, which is a true statement, but I think the practical applications of agile was very often felt much more on the development side of the house than the ops side of the house. I think that's part of the reason.

Josiah Renaudin: I think something that we always have to consider, different companies always have to consider when they're adopting things like ... incorporating DevOps, incorporating agile, is risk. You have to consider, "Of course there's benefits here, but what are the possible issues? What are the stumbling blocks that we can have along the way?" So what risks do teams, especially the operation teams, see in the increased velocity of incorporating agile and DevOps?

Anders Wallgren: I think the first "Oh, crap!" moment that everybody has when they start thinking about DevOps and continuous delivery and continuous deployment, and all of those kinds of things, is: "What do you mean you're going to check in code fifteen minutes into production? What if it doesn't work? What about security? What about manageability? What about accountability?" and all of those kinds of things.

I think those are very right things to worry about, and obviously, traditionally, in going forward, ops is … one of their main goals in life is to keep the lights on and money coming in. Historically that's led, I think, quite a bit to a very risk-averse culture. I think a lot of the processes that were out there before, and to a larger extent are still processed, are kind of about risk minimization or at least; I think some people would say risk minimization theater, which is: "Let's have lots of meetings, let's be very conservative, let's have lots of stamps on everything before we go forward."

I think there are absolutely legitimate concerns and risks that need to be mitigated, but I think the flipside of that is the record shows again and again that a well-crafted, well-maintained, and high-performing DevOps team exposes themselves and their customers to less risks, not more, which as you know it takes a little while to get your head around the fact that if you do something more often you get better at it, not worse. Doing more deployments, more frequently, with smaller change sets ... I think the tide is turning a little bit to where everybody sees the benefits in doing that. Then it's really just a practical question of, "Okay, here are the things we're concerned about legitimately. How do we work those into our pipeline so that we can help eliminate, or at least manage, those risks?"

Josiah Renaudin: Speaking of exposing yourself to fewer risks: Do you think it's easier and safer to start DevOps at a smaller scale, with a smaller team, and then expand it out to the rest of the organization from there, instead of tackling it all at once?

Anders Wallgren: I think you have to start small. I mean, if you're looking at a large bank—one of our customers is one of the top three, or four, or five banks in the country, if not the world—they have literally thousands of application, and so it's not a question of leaving work on Friday afternoon, doing waterfall, then coming in Monday morning and being DevOps, it just isn't going to work.

You have to start somewhere and I think ... you have to cultivate very carefully the grassroots movements that are in favor of DevOps along with the more managerial, executive kind of sponsorship that you need to make that happen on a large scale and in a large company. I think you have to start small. If agile has taught us anything right, it's “divide and conquer”: Break epics down into stories, break stories down into smaller things that you can work on, don't have eighteen-month releases, have two-week cadences. Divide and conquer, and sort of small wins along the way is a very important principle in all of this. Small batches in manufacturing, same thing.

I think over the years that's a lesson that's been hammered into us and for good reason. Start small.

Josiah Renaudin: You just before this said, "You almost have to start small," and I heard the word “almost,” so is it possible to do a companywide adoption of DevOps without really shaking the foundation and messing-up the organizational chemistry? Is that shift just suddenly being like, "All right, everyone's doing this now." Is that just too great?

Anders Wallgren: It's difficult, let's put it that way: I think you need leadership from the top and you need leadership from below. It isn't going to roll out across an enterprise with dozens, or hundreds, or thousands, of applications without significant support from above and from below. So I think it's a question of ... and, honestly, in some cases the organizational chemistry needs to be messed-up, it needs to be rebuilt.

We don't necessarily need to have a cultural revolution or anything like that, but I think we have to acknowledge that, to some extent, the organizational specialization that we've had—that's probably a kind word for it; some people, including myself, will often use the word "silos"—have kind of cut us off from each other a little bit and causes us to lose focus on the big goal that we all have, which is creating value for our customers so that they either give us money, or good things, or good reviews or what have you, or eyeballs for advertisers in return.

It's very common that you get yourself into patterns where everybody's doing local optimizations on the part of the problem that they can see, but we're missing out on the big global optimizations that we can do to really turn organizations—specially legacy-oriented organizations—into high-performing DevOps organizations.

That isn't something that's going to happen overnight and I think part of the really interesting thing that we're seeing with some of the very large organizations that we work with is that they understand that and so they are willing to start to experiment, either with a Greenfield application or, with a particularly forward-looking team that wants to change the way that they're doing things, and then learn from that and let the mistakes happen at smaller levels until you learn a little bit and are more willing to make the mistakes at a larger level, or at least take the risks on a ... larger level, but also get the benefits at a larger level.

I think, yes, we're going to have to mess up some organizational chemistry in the way that “you can't make an omelette without breaking a few eggs,” but all in the name of getting everybody to work better together, not just for the sake of messing it up.

Josiah Renaudin: Yeah, we've kind of established here that it's better to take it at a small scale, go step-by-step, instead of doing everything at once, but even doing it that way can have its certain complications. For example: TechWell, the company I work for, when we decided we wanted to go remote, we wanted to do more of a virtual office, we went team-by-team and it was very simple, where it's like, "Okay, publishing, you start, you're the guinea pigs." Then you go, "Next department, next department." But it's not always going to be that easy with these large complex organizations.

What have you found to be some of the top obstacles for DevOps transformations in these larger, more complex, more massive organizations?

Anders Wallgren: I think one of them is a little bit obvious, but it bears mentioning, which is: Nobody really knows all the details of everything in large companies and large organizations, and so it's pretty impossible up front to know all of the things that you're going to get wrong and know all the things that you don't know yet. You just have to accept that, because otherwise you're just going to be a deer in the headlights and never get started.

Then, in larger organizations ... I think in any organization going DevOps, or agile, or lean, or whichever piece of the puzzle you want to tackle, is cultural change. People are going to have to change, in some cases the way they do their day-to-day jobs, and that can be frightening to people. Whether that's a ten person company or a hundred thousand person company, it's unsettling to people when that happens, but I think in the end there's enough evidence and good stories out there to demonstrate that this works and, in the end, you end up with happier customers and happier employees as well when you make these changes.

Oftentimes I think it's a question of, "Where do we start?" that can be kind of tricky, and then, frankly, navigating the politics of large organizations. There are people who are betting their careers on DevOps transformations and that's a big deal; you may not have your job in a year or two if it doesn't go well. I think that's part of it too, the stakes are a lot higher for some of the stakeholders and that ... If you get somebody who's seen it work and is confident in the outcome they can be a great motivator and leader for that transformation, but if you fall prey to the usual, "'no' is often a safe answer" kind of situation, then that's very difficult to overcome.

Josiah Renaudin: On a similar track here: After you introduce DevOps to the smaller chains, like we were talking about, how do you really harden your practices and spread them throughout the entire organization? Because you can say with their smaller team, "Hey, it worked here, everyone else should try it," but it's more complicated than that, right?

Anders Wallgren: Yeah, much more complicated. I think first you have to keep in mind that you definitely want to, on a big picture perspective, as much as possible you want people to use the same best practices, the same best of breed tools, best of breed processes and all of those things. You have to be very careful not to try to fit every square peg into a round hole, you have to allow for variances. There are going to be different requirements of different teams and different products, or different services if you're doing a microservices approach to that.

I think part of it is you have to be very good at spreading the news and doing a little bit of marketing of what you're doing and making sure that the rest of the organization is aware of it, and this doesn't become one of those, "Oh, well, that's the advanced development team, they do kooky stuff but it's not going to work for the rest of us," or, "Well, you don't understand my requirements, this can't possibly work." I think you have to work much harder to sell a little bit and also to understand and listen to the problems that people bring to the table, because there will be lots of different problems brought to the table and they're not all going to fit into one neat basket that works for three or four, or ten, teams that you start with, so you have to sort of understand that.

I think larger organizations are doing all kinds of interesting things, they'll have a ... They'll set up a team whose goal in life is to interact with product teams on a long term basis, weeks or months at a time, to really get them going on these things, or they'll host a dojo where teams can come in and spend several sprints or several iterations working closely together with people who've done this before and learn by doing as opposed to learning by going to a class or learn by reading, or those kinds of things.

I think when you ... the benefit when you're a larger organization, frequently, is that you have more resources and so you're able to do some of those interesting things that smaller companies aren't necessarily free to do because there are more resource constraints. You have advantages as well as some disadvantages when you're a large enterprise, but I think you have to have a little bit of a plan for how you plan to spread the good news to the rest of the world and how you're going to listen to their concerns and work with them to make things better.

Josiah Renaudin: We've talked about large enterprises, we've talked about small enterprises, and I think it's always just important to note that every single situation, every team is different. It's so hard to have a conversation like this and be like, "All right, that solves everyone's problems, it's all easy now, everyone's got the blueprint," but I do think there's always a middle ground where we can say, "This is at least a good place for you to start no matter what size you are, no matter your makeup," so more than anything, what—generalize; I don't mean that in the negative term of, like, "Here's the basic," but, like, what more ... What's kind of a way to summarize this, this piece of advice that you would give to a company that's coming to you for advice on how to successfully scale agile and DevOps within an enterprise?

Anders Wallgren: It's funny, I think probably the most common question that I get is, "Where do I start?" so I think I'll answer that a little bit and I'll say: I think where you start off, you need to understand your process end-to-end. You need to, as an organization, understand from laptop to live, or room to tune—or whatever sort of analogy you want to use—how your code travels from the time that it's submitted to the source code control system, to when it's live on the site. Of course, if you have multiple products and multiple applications, you're going to have to do that multiple times.

Wherever you start, you have to start with a pretty firm understanding and kind of have all the stakeholders in the room, or in rooms, to figure this out and understand, end-to-end, what your process is. Only then can you start to—especially when you're just getting started—can you start to understand where are some of the first winds, where are some of the most painful points, or where are some of the things that I see now that I see the whole thing end-to-end that I couldn't see when I looked at every individual piece together.

So I think where to start is to have a good, fundamental, honest, frank understanding of what your processes are today and where some of the pain points are, because those are the things that you're trying to fix first of all, but they're also opportunities for ... Low-hanging fruit and easy winds and all of those things are good, and sound a little bit trite, but it's actually pretty important to focus on that a little bit, especially when you're getting started, to prove to people that this is going to work and get them a little bit of confidence and a little bit of swagger, that things can actually get better.

One thing I think we have to understand is that, especially in larger organizations, a lot of times things happen through brute force and heroic effort. Right? Jez Humble has a great thing he talks about when he talks about metrics: He says that probably one of the most important metrics you should talk about is, "How many hours, outside of business hours, do I have to spend to get this thing to work?" And the answer should be “zero.” Obviously, yes, things will happen outside of business hours, you may have regulatory things or we have to do things after trading stops for the day and all of those things, there always will be, but you really have to step back and look a little bit at some of the misery that's required in order to get these things to work with the way that things are being done today, and use that as motivation for making things better tomorrow.

Josiah Renaudin: Thank you so much for talking to us today, Anders. I really appreciate you covering DevOps, because I feel like DevOps is one of those things that any time someone's like, "You should start talking about blank with these interviews," DevOps is top two. Really appreciate you giving insight and hopefully looking forward to hearing more from you on different topics on the future.

Anders Wallgren: I do too, thank you very much.

Anders Wallgren is chief technical officer of Electric Cloud. Anders brings with him over twenty years of in-depth experience designing and building commercial software. Previously, Anders held executive and management positions at Aceva, Archistra, Impresse, Macromedia (MACR), Common Ground Software and Verity (VRTY). Anders holds a B.Sc from MIT.