From 1 release a month to 6 per day

Dale Humby bio photo By Dale Humby
Presented to Scrum Users Group of South Africa (SUGSA), telling Nomanini's story of how our dev team has optimised our continuous delivery process to increase releases from about 1 per month to more than 6 per day.

In summary:

  1. As a start-up, Nomanini has limited cash runway, and is in a competitive market. Need to develop product as fast as possible.
  2. Tooling and automation have a compounding return on investment.
  3. In 2011 we were using Scrum. As we started supporting a production system we had more ops and less dev. We struggled to find time to work on new features. Business and customers got frustrated with engineering.
  4. Story points per sprint (week) was a terrible metric: too much variability to use for reliable estimates. Using average points per week is silly because you will miss you sprint goal about 50% of the time. (Because you get through fewer points than the average.) Not great for building confidence.
  5. Scrum was designed with 1 month sprints in mind. Variability is less over longer time horizons.
  6. There is pressure for shorter and shorter sprints to remain 'agile.'
  7. What happens if we have 1 story sprints, with a release at the end? You get kanban. Independent teams/pairs/people can work on independent stories and release independently.
  8. Kanban uses committed to SLA's and different classes of work = deadline per story.
    1. Fast-track for production issues: Release work within 2 days, 80% of the time.
    2. Fixed delivery date: Release committed to work by a fixed deadline, 80% of the time.
    3. Standard: Release a feature in to prod within 5 days of starting, 80% of the time.
    4. Background: For minor features or bugs, refactors or dev spikes. Release within 5 days, 80% of the time.
  9. We track how long it takes work to travel through our development pipeline. Before we started developers thought bottleneck to get work done was them. Proved it was the release process (test, demo, user acceptance testing, roll-out.)
  10. We built tooling to improve the test, demo, UAT, roll-out process.
  11. We dont estimate individual stories other than ask ourselves: "Is there a good chance we can get this done within 5 days? (our SLA.)" 
    1. If no, then try make story smaller, but still keep business value. (Reduce scope, split in to multiple independent pieces of work that can be released independently.)
    2. Or if you can't split the work, the accept that, for this story, you will miss the SLA. That's why the target is 80% and not 100%.
  12. For estimation: Using a history of work, you can see the trade off between accuracy of estimation and number of days to complete work:
    1. 60% chance that work will be done in 4 days.
    2. If you want 95th percentile confidence, then quote 9 days. 
    3. If managers have a "immovable deadline" then they can what chance they have of hitting it.
  13. For release planning: See there is a 50% chance of completing 11 stories in a month, and 80% chance of completing 7 stories in a month. By looking down backlog can see where will be in 1 or 2 months.
  14. Assumes that the type of work that we do this month is similar to the type of work that we did last month.
  15. We have built tools that 
    1. manage automated testing, 
    2. feature flags to decouple release of software on to servers from release (switch on) of a feature.
    3. Track version through the development pipeline.
    4. And use statistical methodsd to detect issues in production.