The Sauce Journey - DevOps At Scale

Posted Mar 23rd, 2017

In the life of every startup there comes a point, or at least you hope there comes a point, where you need to grow and scale your team. When Sauce Labs closed our last round of funding, we were able to bring on new team members to expand our skills base, increase our project capacity and velocity, and streamline our overall operations. Unfortunately, adding team members doesn't proceed in a linear fashion in terms of adding complexity to a DevOps organization - 10 team members + 10 more, for example, doesn't add up to 20 who will all have the same needs and contributive capacity from a management perspective. Instead, each new team member multiplies the complexity of an organization by something like a factor of 10, which, in this example, works out to a 1 followed by more zeros than looks good in print. The point is, growth isn't a matter of simple addition when it comes to DevOps, but a matter of scaling. Scaling means expanding your operation hierarchically to match the geometric expansion of complexity that comes with new team members, which effectively means hiring more Engineering Managers who can act as intermediaries between engineering and enterprise leadership, and the Scrum teams. When you introduce this mediating layer, though, you face a new problem: having spent so much time and effort on nurturing a culture based on Scrum and DevOps, how do you maintain that culture when you pass the baton of direct management to others? Two methods I've found to be very effective are bringing in coaches who can provide an outside perspective, and using games to generate an understanding of how teams should function.

When we grew from a team of 100 to 300 at Citrix, we recognized that we had to have a continuous process of education in place to help new managers and team members become part of the DevOps and Scrum culture. That meant having dedicated Scrum coaches who could introduce a more formal, structured approach to how teams worked within the parameters of Scrum, and who could provide an external perspective on their day-to-day activities. Sometimes this coaching could be as simple as inviting the teams to talk on a particular topic that's relevant to their activities; since Scrum is largely about communication and information, a conversation can be an effective way to model how a team works together. Every quarter we would bring in a coach who could work with new people to help them get into the DevOps mindset, and who could bring fresh insights to the old hands.

Educating new team members is one half of the challenge with scaling; the other half is making sure that your experienced team members don't carry calcified practices into new parts of the organization. Ceremonies like stand-ups and retrospectives always run the danger of becoming pro-forma, more like rituals that have lost their meaning but are performed to appease distant gods, than like occasions of social interaction. When this happens, games can help teams understand the substance of ceremonies by providing them with an abstract model of their purpose. For example, in one game, you sit four to six people around a table. The team has the goal of completing a task, and perhaps seeing how many goals they can score in a defined period, and each team member has one task to contribute to the goal. The catch is that one individual task is designed to impede or block the goal. The game then becomes about the team identifying the block, working to overcome it, and hopefully learning lessons about communication and interaction that will carry over into their ceremonies.

Regardless of whatever particular method you use, the ultimate goal with scaling a DevOps organization based on Scrum should be to imbue everyone with a sense of ownership over their projects and activities. If you scale an organization and its members feel that they are only carrying out orders from above, then you haven't increased your innovative capacity by adding to your teams, but killed it. The key to successful scaling is not to focus on integrating people into the organization, but refashioning the organization so its members feel that they are primarily responsible for their own activities, and are participants in the larger enterprise.

Joe Alfaro is VP of Engineering at Sauce Labs. This is the ninth post in a series dedicated to chronicling our journey to transform Sauce Labs from Engineering to DevOps. Read the first post here.

Written by

Joe Alfaro