Measurement

These practices measure an operating model to decide if it's worthy of further investment. The cost effectiveness of business outcome protection is an informative, actionable measure of an operating model.

Leading and trailing measures can be used to gauge progress in deployment throughput, service reliability, and learning culture. It's important to select measures that are holistic, actionable, and do not create the wrong incentives. See Measuring Continuous Delivery by Steve Smith.

These practices are linked to our principles of operating models are insurance for business outcomes and operating models are powered by feedback.

Measure deployment throughput

Measure the deployment throughput of digital services as:

Measure	Type	Description	Suggested Implementation
Loose Coupling	Leading	Is a product team able to change a digital service without orchestrating simultaneous changes with other digital services	Inspect deployment history. Calculate how many deployments of different digital services occurred in lockstep
Pre-Approved Changes	Leading	Does a product team have permission to pre-approve its own change requests for low risk, repeatable changes	Inspect ticketing system. Calculate how many digital services have pre-approved change request templates
Deployment Lead Time	Trailing	How long does a release candidate for a digital service take to reach deployment	Instrument deployment pipeline. Calculate days from production deployment back to build date
Deployment Frequency	Trailing	How often are digital services deployed to customers	Instrument deployment pipeline. Calculate rate of production deployments

Measure service reliability

Measure the reliability of digital services as:

Measure	Type	Description	Suggested Implementation
Failure Design	Leading	Does a digital service include bulkheads, caches, circuit breakers	Survey product teams. Are they confident they can cope with downstream failures
Service Telemetry	Leading	Does a digital service have custom logging, monitoring, and alerts	Survey product teams. Are they confident they can quickly diagnose abnormal operating conditions
Availability	Trailing	What is the availability rate of a digital service, as a Nines Of Availability e.g. 99.9%	Instrument digital services. Calculate percentage of requests that are fully or partially successful
Time To Restore Availability	Trailing	How long does it take to re-attain an availability target once it has been lost	Instrument digital services. Calculate minutes from loss of availability target to re-attainment of availability target
Financial Loss Protection Effectiveness	Trailing	What amount of expected financial loss per incident is protected by a faster than expected Time To Restore Availability	Instrument digital services and incident financial losses. Calculate maximum incident financial loss, up to time to restore allotted by availability target. Calculate percentage of maximum incident financial loss protected by a time to restore faster than allotted by availability target

Financial loss protection effectiveness is a measure we've used with customers to gauge the cost effectiveness of You Build It You Run It at scale. It's a check of projected financial loss per incident that is unrealised, because the actual time to restore is faster than the projected time to restore. It's a comparison that can be made between digital services, product teams, and even operating models.

In the earlier furniture retailer example, there was a furniture retailer example with a bedroom digital service. It had a maximum financial exposure of $200K per hour, and a 99.0% availability target.

Software Service	Maximum Financial Exposure In An Hour	Availability Target	Tolerable Unavailability In A Week
bedroom	$200K	99.0%	1:40:48

Assume the bedroom service is unavailable for 30 mins, and the incident financial loss is calculated as $100K. The tolerable unavailability per week for 99.0% is 1 hour 41 mins, which would have produced a $336K incident financial loss. Financial loss protection can therefore be calculated as 70% of a projected incident financial loss was unrealised, as $236K of a theoretical $336K was protected by a faster than expected time to restore.

Software Service	Incident Duration	Incident Financial Loss	Incident Financial Loss At Tolerable Unavailability Per Week	Incident Financial Loss Protection	Incident Financial Loss Protection Effectiveness %
bedroom	0:30:00	$100K	$336K	$236K	70%

For more on measuring financial loss protection effectiveness, see our case study on How to do digital transformation at John Lewis & Partners.

Measure learning culture

Measure the progress of a learning culture as:

Measure	Type	Description	Suggested Implementation
Post-Incident Review Readers	Leading	How many readers does a post-incident review for a digital service have	Instrument post-incident reviews. Calculate how many unique visitors each review receives
Chaos Days Frequency	Leading	How often are Chaos Days run for a digital service	Instrument Chaos Day reviews. Calculate rate of review publication
Improvement Action Lead Time	Trailing	How long does an improvement action from a post-incident review for a digital service take to be implemented	Instrument post-incident reviews. Calculate days from improvement action definition to completion
Improvement Action Frequency	Trailing	How often are improvement actions implemented for a digital service	Instrument post-incident reviews. Calculate days between implementation of improvement actions

Trends in post-incident reviews are of particular interest. Usage in training materials and citations in internal company documents could also be included. See Markers of progress in incident analysis by John Allspaw.

You Build It You Run It on the Xinja banking platform

This customer was a fintech startup, with very limited money and skilled employees. It was a greenfield project in every sense, including business processes as well as technical implementation. I worked on the single technical team responsible for delivery, and therefore the onus of the entire system was on us. It was the classic case for You Build It You Run It. To achieve this, automation was key. Adoption of Continuous Delivery enabled us to focus our energy on development, not deployments. All our changes were tested and deployed through a single automated pipeline, for simplicity and ease of tracking. Operational tasks needed to be in code as well, to allow for seamless deployments of infrastructure and application changes. Monitoring and system reliability were at the heart of the system design as well, since reduced issues meant more time for development. Code quality and system health awareness checks also steered the interests of our team. Since the business and operational knowledge was contained within our team, the customer only needed to contact us to raise their issues. This meant issues were resolved rapidly, and learnings from the issues were fed back into the design, testing and monitoring of the system. Adrian Ng Developer EE Australia & New Zealand