Measuring Success Revisited

“It is wrong to suppose that if you can’t measure it, you can’t manage it – a costly myth.”
– W. Edwards Deming

One of my clients recently asked, “What value is Agile bringing?”
Other clients have asked,

• “How will we know if we are succeeding with Agile?”
• “What metrics can we use to evaluate the progress of our Scrum teams?”
• “How do we measure business value?”

This reminded me of a blog post I wrote almost two years ago, “Measuring Success”. I knew enough then about Agile to look beyond the traditional triple constraint metrics of scope, time and cost for project success. My premise at the time was that “Success” went beyond whether Agile itself was successful. What does that even mean?

Success shouldn’t be about doing or even being Agile.

It should be about the “value” created and delivered to all stakeholders.

• When customers are willing to pay for your service or product then value exists.
• When employees and staff stick around and grow their capabilities then value exists.
• When your suppliers become partners then value exists.
• When the market moves your valuation higher then value exists.
• When your local communities benefit then value exists.

The point is, it is not about running an Agile experiment. It is about value creation and delivery.

Since then, I’ve come to appreciate some subtle, deeper nuances. Perhaps it’s just semantics but it has given me deeper insights. Insights that come in handy when helping organizations introduce Agile ways of working. Here’s a couple of examples.

• The distinction between a metric and a measure.
• The distinction between success and momentum.

The terms “metric” and “measure” are often used interchangeably. In fact, the Merriam-Webster dictionary lists each as the synonym of the other. Digging deeper into the meaning and application of each word, I’ve derived the following distinctions.

• A metric is a standard of measurement whereas a measure is a context sensitive metric.
• Metrics tend to be objective whereas measures tend to be subjective.
• Metrics relate to resources whereas measures relate to people.
• Metrics are to finance as measures are to happiness.

Why do I distinguish between “success” and “momentum”?

• Success implies a final outcome whereas momentum denotes movement with growing strength.
• Success is binary in nature. You either succeed or fail. We have been conditioned to see success as good and failure as bad. Momentum accounts for the ebb and flow of life. It builds off adversity. Like the classical push-back shift in hockey or rally run in basketball.
• Success appears linear and unidirectional. Momentum meanders and adapts.
• Success is a destination. Momentum is a journey. With success what happens after the denouement? With momentum, energy builds and carries on.

So, should we pick success metrics or measure momentum?
Why choose?
Why not both?

With systems thinking in mind, imagine a collection of entities impacted by the introduction of Agile ways of working. Starting with individuals on the far left moving through teams, lines of business, the enterprise and ending with the customers on the far right. We could adopt a (mostly) “measure to the left” and (mostly) “metrics to the right” stance.

 

Metrics to the Right

Towards the right, success for customers using your products and services include the traditional metrics of:

• Customer Satisfaction Score
• Net Promoter Score

And the emergence of fit for purpose measures as introduced by Alexei Zheglov and David J. Anderson. A holistic, customer-centric fit for purpose (F4P) framework with fitness criteria that include:

• Time to Market
• Quality
• Replenishment Frequency
• Release Frequency

Success for the enterprise often follow universally standard financial metrics and ratios such as:

• Revenue growth
• Earnings Per Share (EPS)
• Earnings Before Interest, Tax, Depreciation and Amortization (EBITDA)
• Free cash flow (FCF)
• Return On Total Capital (ROTC)

With start-ups that don’t have established financial metrics, then Innovation Accounting as introduced by Eric Ries in his book “The Lean Startup” may be a viable alternative. With innovation accounting, what’s important is not the raw numbers but rather the direction and degree of progress. Examples of innovation accounting metrics include:

• Profitability of each customer
• Cost of acquiring new customers
• Repeat purchase rate of existing customers

Measures to the Left

Towards the left, we’re dealing with people. How each of us feel can’t be standardized and will change from day-to-day. The best we can do is to use our senses and measure the buzz in the air. How is everyone feeling? How are team members behaving? On their own ? With each other? With management? Is the team gelling? Is momentum gaining or dissipating? Nothing more sophisticated than our God-given senses is needed. Ways to help surface those feelings include:

• Noki Noki calendar
• Core protocols like check-in
• Micro surveys like Doc Norton’s Software Joy metric
• Dixit retros harnessing the power of pictures over words
• Team health checks

In the Middle

Teams and lines of business have traditionally been assessed against the triple constraints of projects. Agility metrics such as:

• Customer Lead Time
• Work In Progress (WIP)
• Throughput or Delivery Rate
• Cumulative Flow Diagrams (CFDs)
• Run Charts
• Flow Efficiency

bring the customer into sharp focus. They remove silos by horizontally aligning teams and the organization towards end-to-end value creation and delivery.

The short game on the right maximizes the money.
The long game on the left maximizes the people.

You need both.