Skip to main content

A DevOps Transformation in a day

05 Mar 2019

| Author: Paul Dykes

Share

A DevOps Transformation in a day

DevOps offers many benefits to organisations, including enhanced customer responsiveness, faster lead time from commit to deploy, and an improved ability to scale and pivot in response to market changes. So, why isn’t every organisation using DevOps practices?

Among other reasons, DevOps is at least as much about cultural change as technological change. Transformations can face internal resistance – it may be that people are concerned that governance will be reduced or that roles might become redundant. And it is important for the transformation to be organisation-wide; a DevOps transformation that is contained within the IT organisation will have little chance of succeeding.

Help is at hand. IndigoBlue’s DevOps consultancy helps organisations to chart their best route to DevOps success and, in partnership with G2G3, we were pleased to offer a DevOps Simulation event in London on the last day of the February heatwave.

How the simulation runs

This day-long training course uses experiential learning to create breakthrough understanding around the positive impact of DevOps, aiming to reduce cultural and political barriers around organisational change. Focused on culture, people and processes, the course simulates the way work flows through an organisation.

In three rounds of planning, delivery and review, delegates go on the journey of the first year of a DevOps transformation. Working for an imaginary retail organisation, they are tasked with introducing new features to online and in-store systems while maintaining live systems which process customer transactions.

No actual software is developed. Development is simulated through making Tetris-like shapes out of card; incident resolution by Operations is simulated through answering Mensa-style questions. The outputs are fed into a software application which provides a live dashboard, monitors delivery throughout each round, introduces service incidents and provides analytics on performance. These analytics are discussed in the review, encouraging the teams to consider how they might improve the way they work in the next round.

Forming the teams

After an introduction and welcome from IndigoBlue’s Principal DevOps Consultant, Bruce Richardson, the simulation got underway with G2G3’s Jaro Tomik inviting delegates to form into the following teams:

  • The Business – tasked with prioritising features to be developed and reporting incidents to the Service Desk
  • Scrum Master/Product Owner – tasked with managing the backlog, monitoring development and ensuring quality
  • Development (two teams) – tasked with developing new features
  • Testing – tasked with checking that fixes and new features work
  • Operations – tasked with deploying new features, maintaining systems and resolving incidents
  • Service Desk – tasked with reporting incidents and logging tickets for Ops to work on

In their day jobs, our guests’ roles include management, software engineering, operations management and DevOps engineering. Each guest was invited to adopt a different role from their day job, a deliberate tactic to give delegates the chance to gain an understanding of the challenges faced in other teams and what they require in order to optimise flow and deliver value.

Round 1

Planning

Each team was given specific instructions to read before delivery began, and then Round 1 got underway. After 25 minutes of sporadic and confused activity, and a deluge of service incidents to contend with alongside feature delivery, the team came together for a retrospective. 

Chaos had reigned, but why?

DevOps-Simulation-February-2019-Waterloo-London-lo-res-5.jpg

Review

  • The Business team admitted that it had put too much into delivery at the same time.
  • The Product Owner and Scrum Master could not prioritise or monitor work in progress.
  • The Development teams’ workload varied wildly, they lacked clear priorities and neither team knew what the other was working on. Along the way, they developed a feature for deployment on a platform which another team knew to be no longer operational – a fact they had not communicated beyond their silo, wasting time and money.
  • Operations had self-optimised, dividing work into Release and Incidents, each assigned a lead to co-ordinate work using a basic kanban board. However, they lacked visibility of how much incidents were costing the business and found the biggest priority incident difficult to resolve.
  • Meanwhile, Service Desk felt overstaffed as they were just passing work on without priorities.
  • The Test team had little to do until right at the end of round.

Results

This chaos was reflected in the analytics. Only two out of seven new features were deployed and only 19 out of 96 possible customer transactions were successful. 18% of forecast revenues was achieved, there were five SLA breaches and availability was 81% – way below the 99.9% availability recommended by ITIL.

The Business rated the group’s performance at 2/10.

Round 2Planning

Planning

After a bruising first round, everyone was keen to plan and make improvements. The teams discussed how best to organise their work and how to work better together. In Round 1, work had been processed in large batches in a waterfall way, and there had been minimal communication between silos. In Round 2, the teams would aim to ‘shift left’, discussing issues and constraints as early as possible.

The Business would articulate value clearly to inform prioritisation and so that every team would understand why one piece of work was more important than another. The Product Owner would define acceptance criteria to ensure the intended value was delivered. The Scrum Master and Product Owner would track all work in progress, both incidents and new features.

DevOps-Simulation-February-2019-Waterloo-London-lo-res-7.jpg

The Development teams would be merged. Operations would create a ‘code repository’ of proven solutions that could be reused and continuously improved. The Service Desk would have a strengthened role, providing shared visibility to help the business and teams prioritise incident resolution. And more automation would be introduced before the end of the round.

So, what happened?

Review

  • The perfect example of a J-curve. The teams were trying out new processes and new ways of working for the first time – and this was both difficult and yielded poorer results than expected. In the real world beyond the simulation, this is often the point at which organisations give up and conclude that DevOps isn’t right for them.
  • The Business had felt more aware of the needs of other teams and the impact of their decisions. But planning had taken too long, so low value work was implemented first just because it was ready to go – the highest value was deployed fourth.
  • The Scrum Master and Product Owner found tracking work challenging; work items got lost and sometimes they found themselves tracking things that had already been signed off.
  • Development found themselves doing quite a bit of rework, which they assumed to be issues with acceptance criteria; it transpired that they had not been continuously improving and had continued to use old documentation.
  • Operations discovered that a service had been released in Round 1 of which they were unaware and about which they had no information on how to support it. More significantly, Ops did not communicate clearly to the Business that a certain incident involving an apparently low-value system was in fact a hack – as such, the hack was not prioritised for urgent resolution.

DevOps-Simulation-February-2019-Waterloo-London-lo-res-8.jpg

Results

The results were only marginally better. Only 22 out of 96 possible customer transactions were successful and there was a $300K overspend on new features. A few more features were released to production, but it was clear that the Testing team was now the bottleneck.

The Business rated the group’s performance at 5/10.

Round 3

Planning

Determined to do better in the final round, the group agreed that not only was better planning needed, but it should be continuous planning and prioritisation – when an incident arose, it needed to be prioritised in the context of everything else currently in the pipeline.

Ops and Test would work more closely and proactively, planning which environments to deploy each feature to while it was still in development, and automating as much as possible. Adopting Lean practices, all the teams would pull work into delivery rather than having it pushed on them. The Service Desk would carry out first-line fixes but also enable teams to resolve things themselves.

During Rounds 1 and 2, issues had recurred which was suggestive of underlying problems – so, Ops would work on upgrading services to resolve these, managing a new ops backlog alongside the product backlog and the incident log.

So, how did it go?

DevOps-Simulation-February-2019-Waterloo-London-lo-res-11.jpg

Review

Really well!

  • The Business felt better able to plan and prioritise. 
  • The Product Owner and Scrum Master felt more in control and able to track work in progress.
  • The close collaboration between Development, Test and Operations meant that only one piece of rework was needed.
  • By using its code repository to accelerate work on deploying new features, Ops freed up time to work on service upgrades.
  • The Service Desk was busier but had the information it needed to play a pivotal role, including identifying high value, recurring issues that could be fixed. 

DevOps-Simulation-February-2019-Waterloo-London-lo-res-15.jpg

Results

As a result, a large number of new features was deployed while keeping services up and running, resulting in revenue of $27M out of a possible $35M. Analytics showed that the same amount of time was spent on Development, but much less time on testing. Thanks to this improved productivity, features went live quickly, in order of business value and then reached the point at which they delivered ROI sooner; the highest value feature started delivering ROI nine minutes into the 25-minute round. 

The Business rated the group’s performance at 9/10 – there’s always room for continuous improvement in DevOps!

Asked to explain what had made the difference, the delegates pinpointed a range of factors, including: improved knowledge sharing, collaboration and communication; continuous planning and prioritisation; automation, continuous integration and continuous delivery. 

The event received overwhelmingly positive feedback and if it sounds like it might benefit your organisation, please register your interest here in participating in one of our upcoming simulations.

From simulation to reality

Bruce rounded off proceedings by explaining that while planning a DevOps transformation can be daunting, at IndigoBlue we have charted the routes that organisations can take from traditional software management to DevOps best practice. We have set these out in The DevOps Pathfinder which you can download here:

Download The DevOps Pathfinder

 
And you can watch an overview of The DevOps Pathfinder here:

DevOps Consulting

DevOps consulting

Find out how our consultancy helps your organisation scale up with confidence to meet ever-growing business and customer demand.

Find out more

Subscribe to receive our email updates

Author

Author profile image

Paul Dykes

Paul is a highly experienced digital product owner and digital marketer, with experience of leading digital marketing teams and directing the development of multi-million-pound web development projects.

CONTACT US

How can we help your organisation?

+44 (0)20 7692 4832

How will we use the information about you?

When you complete this form and submit your details, you are trusting us with your personal data. Our Privacy Notice informs you of what personal data we collect, why we collect it, and of your rights in relation to your data. By pressing submit, you indicate that you have read and accepted the terms of our Privacy Notice and that you consent to our processing of your personal data as described in the Privacy Notice.