Agile as a project approach enables value creation whilst also empowering sponsors and senior stakeholders to effect governance over the investment. However, this depends on effective mechanisms for translating the team view into a stakeholder view of progress, risk and opportunity. This paper identifies the challenges and solutions in achieving this translation.
Origins of the Challenges
The adoption of an Agile approach to product development and delivery is fundamentally concerned with incremental delivery of value through collaboration between team members. It is therefore different to the structured and process-heavy Waterfall approach, where the development and delivery lifecycle erects barriers between different teams for each lifecycle stage.
There are several factors that contribute to this frustration and mistrust that need to be recognised and understood by both the project team and senior stakeholders
Organisations that have traditionally used the waterfall approach have developed structures and processes for the management and governance of both project and operational activities. These are based on alignment between the operational management of capabilities and project lifecycle stages that map to specialisms such as business analysis, technical design, development and testing. The transition from Waterfall to Agile challenges these structures and processes and requires change at all levels.
Conversely, Agile team culture has a strong inward focus that often leaves senior managers feeling uncomfortably distant from what is happening within projects and unsure of how to create the opportunity for team success whilst feeling confident that they can fulfil their management and governance accountabilities.
The Agile Governance Challenge
Governance of Agile projects has become a significant topic. There is a growing demand from organisations that are running large Agile projects for external review to seek clarification of status and reassurance that the projects are under control with appropriately managed risk. These reviews often find that there is significant discrepancy between on-the-ground status and senior management perception.
IndigoBlue has helped many organisations with transformation to sustainable Agile adoption. Many of these organisations have previously either encouraged Agile adoption but lacked confidence to apply Agile to larger-scale or business-critical projects, or they have treated them as exceptional and have not formalised Agile as part of the development and delivery process.
These scenarios have a common theme – there is frustration, lack of understanding and mistrust between the Agile teams and senior stakeholders.
There are several factors that contribute to this frustration and mistrust that need to be recognised and understood by both the project team and senior stakeholders. When these factors are understood and accommodated, the opportunity for sustainable and successful adoption of large scale Agile projects can be achieved.
The high level of uncertainty that exists on an Agile project is mitigated through low-level team activity. A Waterfall approach attempts to remove uncertainty through detailed analysis at the start of the project. Where it cannot be removed, assumptions are made and validated at lifecycle stages, with formal sign-off and gated processes. This lifecycle structure gives the impression of certainty and control over requirements, solution and quality in consecutive stages. However, this certainty is often mistaken, with projects requiring significant rework or greatly overrunning estimates and timescales.
The uncertainty carried through Agile projects means that scope and/or solution is not fixed; however, time and budget are. The team is empowered to maximise value using techniques at the team level for evolving the optimal solution and feature set. This requires a different approach to communication with senior stakeholders so that they can be confident that the risk balance is appropriate and that the level of uncertainty is not too large.
Planning and Reporting
Project planning, forecasting and reporting is different between Agile and Waterfall projects. Waterfall is a fundamentally activity/deliverable-based approach that can map onto Gantt charts with stage-based milestones, whereas Agile is a time-boxed iteration-based approach, supported by a product backlog.
Agile provides valuable information at the team level on progress and the challenges for meeting goals. It provides this in a detailed and timely way to support rapid issue resolution at the team level
The Waterfall approach means that different functional units (such as QA and test) can plan activities based on the early production of definitive specifications (e.g. the functional specification). An Agile approach requires a cross-functional team to be committed to a continuous collaborative approach to developing a solution, with iteration-based resource and activity planning within the team.
Unlike Agile, a Waterfall approach enables clear status against a plan to be mapped and communicated; scheduled activities and deliverables are either complete or incomplete. The status for an Agile project is largely based on velocity; the rate at which the product backlog is being addressed. However, velocity – and its associated language of 'story point' – are a team view that requires translation for steering groups and senior management.
The most common Agile processes, such as XP and Scrum, are intrinsically engineering- and team-centric. Indeed, they reinforce an inwardly team-centric focus that views anyone outside the team as a potential distraction rather than as a key stakeholder. Often, upward/outward reporting based on velocity is a casualty of this, leading to senior management demanding traditional project status information, which then leads to increasing tension between the Agile project team and senior management.
There is a belief associated with Agile that because the ‘customer’ is (or should be) embedded in the project team and directly involved in the iterative team process, then the ‘customer’ has clear visibility of the project status and metrics. However, very often the role of the ‘customer’ is to provide understanding of requirements and priorities, and does not own the strategic aim of the wider change programme. Whilst this is an invaluable role, it does not satisfy the needs of senior management to have management and governance accountabilities related to the project.
The effective governance of any project, regardless of the methodology or approach, requires a baseline against which achievements and forecasts can be communicated
Agile provides valuable information at the team level on progress and the challenges for meeting goals. It provides this in a detailed and timely way to support rapid issue resolution at the team level. However, this requires translation for communication at the overall project level and for relevance to a wider portfolio level.
It cannot be assumed that senior management, or indeed anyone not directly involved in team activity, is able to comprehend the detail and have the time to translate detail into significance within a broader context. As the Agile process relies so heavily on face-to-face communication, it can even be hard for team members who are absent for a short period of time (e.g. on leave) to understand what has occurred in their absence.
Whilst it is possible for small projects to have small stakeholder reference points, larger projects usually have a wide range of stakeholders with governance accountabilities. It must not be assumed that the Product Owner, who may be embedded in the team, is the only stakeholder that needs confidence in the project.
For large projects, these stakeholders form a steering group as a focus for managing their accountabilities. This creates a second tier of management that requires aggregation of project status at a snapshot in time, often monthly.
Effective Agile Governance
The effective governance of any project, regardless of the methodology or approach, requires a baseline against which achievements and forecasts can be communicated and against which the consequence of change can be recognised.
Governance also requires a process and conduit for communication. This is most commonly facilitated by a project manager but, on Agile projects, increasingly by product managers.
For the successful governance of Agile projects, the following approach should be adopted. There must be:
- a project initiation stage that delivers a baseline plan for the full scope of the project and a solution baseline
- a process for facilitating changes to the baseline
- a process for governance that defines the governance group, process and necessary input
- a process for progress tracking and forecasting future progress that can be communicated against business milestones
The project initiation stage should be undertaken with a view to minimising delay to development activity whilst providing a clear understanding of:
- the project value proposition and incremental plan
- the scale of work needed
- the timeline for completion, with incremental milestones for business value
- the significance of uncertainties carried forward, along with an outline mitigation plan
- the processes for delivery and governance
These are the minimum criteria for both a successful team and effective governance. The initiation should take a maximum of six weeks, even for large-scale projects.
In any project, change is inevitable and it is therefore essential to ensure that the change process is straightforward to help the team arrive at the optimal solution. From a governance perspective, there must be recognisable criteria for success against which change can be judged for value and/or risk. This empowers the team to agree change whilst at the same time identifying where that change may need to be validated by the stakeholders that have a vested interest in project success criteria.
Over a project lifecycle, there is usually significant (around 40%) churn in requirements between the baseline and the deliverables. When dealing with the changes that make up this churn, it is important to be able to trade similarly scaled items of work based on value, thereby maintaining an overall estimate consistent with the baseline whilst maximising the value created. Where there is significant deviation to the overall estimate, or other serious knock-on effects (e.g. consequential change) then senior stakeholders need to be aware of the associated risk and be able to influence decisions.
The governance process must be outcome-based and not activity-based, with the governance group defining a successful project outcome in terms of business objectives. In order to do this the governance group must be provided with information on risks to the key success criteria (usually value created), cost, time and quality. This requires translation of team metrics to business metrics around progress and risk.
The essential information that must be presented to the governance group is:
- Incremental Business Outcome Milestone Status and opportunity/risk
- Cost and Resource forecasts and opportunity/risk
- Value Outlook
- Other project risks associated, including any increased debt against the solution, quality, sustainability, etc.
In any project, change is inevitable and it is therefore essential to ensure that the change process is straightforward to help the team arrive at the optimal solution
This requires translation from sprint or iteration reporting. At the iteration level, the primary information for reporting is velocity. There must be a mechanism for extrapolation and assessment of milestone, resource and quality risks.
It is insufficient for an Agile project to simply track velocity; other factors such as technical, people and quality need to be monitored for increasing risk to sustainability.
There is no reason why senior management and senior stakeholders of Agile projects should feel unable to fulfil their management and governance accountabilities. The key is understanding an outcome-based governance approach, with acceptance of uncertainties and empowerment of the team, coupled with understanding of what makes a sufficiency of baseline and process.
Senior management need to recognise that effective governance of Agile projects requires change from the traditional governance process and information flows. IndigoBlue is able to provide both a proven Agile framework and the experience to enable such change.