Measuring Success

“Working software is the primary measure of progress.”

– Agile Manifesto Principles

Working Software Is Not Enough

“Working software” is the definitive measure of an Agile software team’s performance. It represents the net result of all the hard behind-the-demos conversations, collaboration and work within the team. It represents where the team is on its journey towards its purpose. It’s tangible. You can see it, touch it and feel it. It generates pride of accomplishment within the team. Product owners, product management and customers can use it like a ship’s rudder to steer the team forward by providing feedback on what’s acceptable and what’s not – on its fit for purpose. But it wasn’t enough for the many other stakeholders in our business that supported and invested in IT’s move to Agile. Moving to Agile conjured up different benefits and expectations for each stakeholder beyond progress of the team. For them success meant different things. Here’s a sample of what our stakeholders asked.

• Finance asked “What savings can you book into the next budget forecast?”
• HR asked “What impact has there been on the morale of the team?”
• Sales asked “How much quicker can we deliver new capabilities to market?”
• Customer Support asked “How are we doing on our existing production issues?”
• Team members asked “Are we getting more customers and making more money?”
• The CEO asked “What can I share with the board on status of your Agile transformation?”

Just as every implementation of Agile is different, the definition of success for every Agile implementation can be different; not only for the business as a whole but for individuals within the business.

The hopes and dreams for our Agile transformation went far beyond “working software”. Often the ends are used to justify the means. But with Agile, success is measured not only by “what” we deliver but also by “how” we deliver it.

And so we needed to align on expectations and the meaning of success.

What’s Our Meaning of Success?

Agile is not a silver bullet. It won’t solve world hunger or the plight of whales (at least not yet). The benefits and implications of Agile can be far-reaching.  But if unchecked, everyone’s hopes and dreams can become hitched to the promised land of Agile leading to misaligned expectations. With the increasing popularity of Agile, the danger of misaligned expectations becomes even greater as people hear of wild success or failure stories at different companies and expect the same results at their own company. Problem is Agile is highly dependent on the culture of a company which will be different from company to company.

Measures of success quantify. But without understanding the meaning and connectedness of success, the measures of success appear as simply a string of incoherent numbers. We needed to align on the meaning behind the measures that resonated with all our stakeholders.

Even when you make all your numbers, you can fail. I knew of one company where their IT shop had measures and targets in place for scope, schedule and quality. They consistently met and at times bettered every target for every release and yet their business stakeholders were still not happy. The problem was the business was gunnysacking a set of unstated expectations that were not covered by IT’s scope, schedule and quality metrics. These included responsiveness, adaptability and collaboration. It wasn’t until IT asked why, despite meeting all of their commitments, the stakeholders were still not happy that these additional indicators of success surfaced.  IT as befits their left-brained engineering stereotype was focused on the delivery specifications. The business was focused beyond the specifications, on the nature of the behaviors and relationships. Expectations were not aligned. From that point on, the IT shop not only measured scope, schedule and quality; they also surveyed the stakeholders to gauge their level of overall satisfaction.  It was a great example of why success is not just about the numbers; of why the means are just as important as the ends.

To align on our meaning of success, we started by understanding how our current baseline capability was perceived and the gaps that needed to be addressed.

• What business value we delivered?
• What mattered? What didn’t matter?
• What was working? What wasn’t working?

We then held crucial conversations with everyone to understand what their personal WIIFM (What’s In It For Me) statement was – their aspirations.

Finally we looked at both the gaps and the aspirations through an Agile lens to come up with our own unique meaning and measures of success.

Measuring What We Mean

Success for us meant three things.

1. Transformation – transitioning from our existing IT development process to Agile
2. Product – delivering high business value to our clients
3. Cycle – staying on track with our cycle commitments

We came up with the following measures to gauge the success of each.

Transformation Measures

• Business Unit (BU) Satisfaction – a set of qualitative questions that gauged our stakeholders overall satisfaction with IT. Questions covered:
  • Speed to Market
  • Cost Efficiency / Productivity
  • Quality
• Agile Team Satisfaction – a set of qualitative questions that gauged our IT team members’ overall satisfaction with IT. Questions covered
  • Personal growth
  • Employee engagement

Product Measures

• Time to Market (TTM) from approval to production deployment
• # of resumes added to our job seeker resume database
• # of job seeker Career Alerts that have been created on our platform
• # of Facebook likes for our platform

Cycle Measures

• Burn-down charts to measure the amount of work remaining at any point in time within a cycle
• Velocity to measure the amount of work completed each cycle
• Quality (Customer Value)
  • # of Severity 1 issues deployed into production
  • Repeated issues with the same root cause
  • Total # of outstanding support issues
• Cumulative Flow Diagram (CFD) to measure trending of work-in-progress (WIP) and value delivery from cycle to cycle

As we succeeded in these areas, our level of Agile competency and confidence increased. This prompted us to raise the bar on our meaning and measures of success so we could not only sustain what we had achieved but also continue to grow. This included adjusting targets on our current measures and also adding some additional measures such as:

• Corporate Culture Survey to measure the impact of our Agile transformation on the overall culture of the company especially as our Agile ways of working started to impact all parts of the business. (Transformation Measure)
• Customer Satisfaction to share in this goal with our Sales business stakeholders to demonstrate our willingness towards shared skin in the game with the business (Transformation Measure)
• Product ROI (Product Measure)
• # of items moved from “Ideas Backlog” to “Product Backlog” (Product Measure)
• # of external interruptions to team and time lost per cycle (Cycle Measure)
• # of days blocked on user stories (Cycle Measure)

As we measured our success, we learned the importance and value of getting behind the numbers collected. This was especially true of the survey responses. Getting additional feedback and context on the responses was another opportunity to engage on issues that mattered with our stakeholders – team members, BU owners, Customers. Achieving clarity through conversation became part of our continuous improvement regimen.