Responsive Deployment

Fred Brooks said, “Faire de la bonne cuisine demande un certain temps” (to make great food requires a certain amount of time), but that’s no reason to let food sit around and get cold or, worse, spoiled. It’s lean to deliver when ready. But, remember, it’s aggravating to have food forced on you when you’re not hungry.

...Sprints are flowing well and the team has demonstrated the discipline to manage its velocity and to predict its deliveries well. The customer base is manageably small, and there is an intimate trust connection between the Scrum Team and the market, as well as between the Product Owner and the Developers. Sprints are as short as practical given business forces, the need for a sustainable rhythm, and the need to balance the trade-off between planning overhead and production time. Business surprises may arise that require attention to market products on time frames as short as a day.

✥ ✥ ✥

Working in rhythm is good, but so is smoothing out flow (elimination of mura (斑)), and reducing the inventory of items waiting for the team to deliver them to the market. Having a good Sprint rhythm is key to the success of Scrum’s short-term planning, but sometimes even a Sprint can seem too long given market urgencies. And the rhythm itself can lead to subtle lapses on the part of the team, which may postpone testing, quality assurance, or other tasks to the “last responsible moment” right before delivery. This, of course, violates Swarming. Furthermore, the Development Team may believe that it’s important to test everything together that will be delivered together. (While there’s a bit of truth to that, it’s more important to test what you can as early as you can.) That may cause the team to push testing to the end of the Sprint. That gets in the way of the team thinking like a true team that takes each item to Done (see Definition of Done) before moving on to the next. It leaves many Product Backlog Items (PBIs)—and the product as a whole—in an uncertain state.

On one hand, it’s nice to be able to give the market a generously sized package of product or functionality at the end of a Sprint. After all, it makes the Development Team look good if the deliverable is large. Development rhythms help the enterprise gain economies of scope by packaging the administrative procedures to deliver a set of features together, rather than administering each one separately. On the other hand, the team could have delivered much of the same business value piecemeal to the market much earlier. An early delivery can provide more timely value to the market and more timely feedback to a feedback-hungry team seeking to fine-tune the feature in the next Sprint. If you truly wait until the end of the Sprint to deliver a feature, it will take at least another Sprint to evaluate how it plays in the market, and a third Sprint to respond to customer feedback. Part of smoothing out flow (eliminating 斑) means reevaluating how frequently to deliver. Worse, product that sits in the warehouse waiting for deployment at the end of the Sprint is inventory, and inventory leads to waste.

Whack the Mole says to deal with bugs when they arise. Yet, if the team can’t deliver product until the end of the Sprint, the market will continue to suffer with the corresponding product defect until delivery takes place. There can be a high opportunity cost in delaying the delivery of bug fixes to coincide with unrelated feature deliveries; yet, there are administrative costs associated with delivering outside the normal product cadence.

Therefore:

Release selected Product Backlog Items to the market as they become Done and as the Product Owner approves them in the course of the Production Episode. All usual Scrum quality standards such as Definition of Done still hold. Ensure that the cost and overhead of piecemeal delivery doesn’t exceed the benefit. Keep as much of the administrative processing (such as invoicing for deliverables, updating the roadmap) on the Sprint cadence as possible.

The Scrum Team can deploy a partial product increment to consumers as soon as it meets the Definition of Done. This can reduce the amount of time that a defect stays live in a deployed product. Most of the time, this technique complements normal Scrum rhythms that support the natural cadence of Follow the Moon, and the team should continue to attend to other important rhythms of the Scrum cycle through patterns such as Named Stable Bases. In a normal Scrum, reserve Responsive Deployment for urgent and exceptional cases. In more advanced situations where the market has a close relationship with the Scrum Team, the team can use this approach more extensively to break the normal Scrum rhythms by spreading release administration effort across days of the Sprint instead of compartmentalizing it in the Sprint Review. This eliminates some mura.

For each partial product increment, the team (or, more likely, the Product Owner and some subset of the Development Team) does a mini-Sprint Review. This short and simple review focuses on the tactical issues of the partial product increment and leaves the strategic issues to the Sprint Review at the end of the Production Episode. These reviews need to be short and focused to not consume too much of the Development Team’s time. The Product Owner team can help this situation by taking on as many of the review tasks as possible.

✥ ✥ ✥

The team still delivers a Product Increment but smooths out the delivery to minimize the inventory of deliverables that are Done but awaiting deployment, and puts urgent deployments immediately in effect.

AT&T used to ship its main software content (called a generic release) on a periodic basis. The company deployed emergency fixes on demand as broadcast warning messages (BWMs) that comprised incremental patches. According to a 2012 article, Facebook ships at least daily, and often twice daily ([1]). More recent articles note that they ship every few hours. [2] The Finnish legalized gambling company Ålands Penningautomatförening (PAF) has an aggressive and disciplined release structure that can turn around changes in hours to respond to market changes (related to Jim Coplien by Ari Tanninen, 18 April 2011).

Key to the success of all of these approaches was that they incurred low per-delivery cost. In all of these cases, the enterprise minimized the overhead of incremental delivery cost with relentless improvement (see Kaizen and Kaikaku) over time. When transitioning to ad hoc delivery schedules, pay attention to velocity (see Notes on Velocity) and other indicators of value such as absence of defects or customer satisfaction, and manage the trade-offs between uninterrupted development flow and immediacy of feature delivery.

Delivering to the market during the Production Episode opens the door for feedback about that partial increment. End users and customers may come to expect the same attentiveness for a Responsive Delivery as they do for a regular delivery at the end of the Sprint. That raises the specter of a full-blown Sprint Review for each partial increment, with prospects for compounded feedback in response to every interim change. Giving priority to such changes takes ever-increasing time away from working on the original Sprint Backlog and Sprint Goal. It may place either in jeopardy. The Product Owner should intercept these interactions and protect the rest of the team from disruption. In most cases, the team should defer discussions about feedback on partial increments to the Sprint Review. The only potential exception to this rule is a business-affecting defect in a previously delivered Product Increment that affects the end user’s day-to-day operations. If an emergent requirement becomes apparent as a result of delivering a partial increment, the Product Owner might consider invoking Emergency Procedure.

Emergency Procedure employs this pattern under more extreme circumstances than one finds in textbook Scrum. In Emergency Procedure, the team can deliver a product in response to an emergent request from the Product Owner, before the end of the Sprint, usually in some highly urgent context that escalates to top management as a “bet the company” high-risk issue. Not all invocations of Emergency Procedure entail abnormal Sprint termination. Responsive Deployment turns the exception into the rule: every day the Development Team works to finish what it can deliver soonest. The team manages itself to these deliveries, often without any particularly urgent interaction with the Product Owner. Often, the team can deliver in the order that best suits planning constraints inside the Development Team. And often, there is enough trust between the Development Team and the Product Owner that even the Product Owner can add solid guidance to the ordering of PBI work within the Sprint. Such trust is necessary for this pattern to work; otherwise, the Product Owner will be seen as disrupting an Autonomous Team. To run a Scrum Team this way—with daily Product Owner engagement—requires a high degree of team maturity.

Product Owner engagement aside: To employ this pattern at all requires that the team have enough discipline to plan well, and that its market and development process be mature and regular enough that delivery before the end of the Sprint doesn’t turn into ad hoc development. The team should have a strong Definition of Done and, perhaps even more so, a strong Definition of Ready. Lastly, the pause to assess the partial product increment prior to delivering it disrupts flow. Good flow is a prominent enabler of team throughput.

For these three previous reasons, this is a “Ha” (破—“break the rules”) pattern in the Shu-Ha-Ri progression: it is outside the normal rules and only mature, disciplined teams should apply it. The pattern should be applied selectively, either in the case where there is a seamless interface between the Scrum Team and the market, or as an infrequent exception to handle urgent situations such as on-demand customer site visits that require the intervention of the team. The point is that the delivery should respond to something, and that delivery should be based on pull rather than push. The overall value proposition—including client comfort—should always be foremost in these decisions. Much development operations (DevOps) hype in software development tilts toward encouraging this behavior to help the development side keep a clean plate. It’s worth exploring the consequences that might arise from a more casual use of Responsive Deployment.

To call this pattern ‟Continuous Deliveryˮ would be a bit of a misnomer, but it remains the term of fashion. Delivery is rarely truly continuous; good product development depends on the deliberation tied to discrete steps. That is why a culture of Named Stable Bases looms important. The vendor and market together must understand where they stand on delivery; the team must be able to step, rather than slide, towards a demarcated Sprint completion. Compare One Step at a Time. Development should proceed in discrete Programming Episodes that are still best managed with Swarming.

Responsive Deployment presumes that the market will absorb and use the product immediately when delivered: otherwise, it is simply a way of removing the product warehousing from the producer to the consumer premise. The lean principle of “just in time” is driven by an expectation deadline, where completing the product early may be just as bad as completing it late (for example, because of client-side inventory costs or because of possibly reduced test coverage).

Most product development reduces the negotiation overhead between developer and consumer by chunking deliveries around agreed upon delivery schedules; that is much of the rationale behind Sprints in the first place. As noted above, pausing for a review during the Production Episode disrupts flow. In the way of metaphor, Responsive Deployment is like fast food: it is responsive to individual clients, with the potential downside that it may introduce queuing delays while one set of customers waits for the completion of other customers’ orders. Sprints run more like a banquet house that can predictably deliver to a large market on a regular basis, though such cadence cannot achieve the time scales of fast food. These forces tend to relegate Responsive Deployment to a place more within the delivery of solutions to emergent requirements than for the normal Value Stream flow. Alternatively, Responsive Deployment might work between a single development organization and single consumer: it may be too difficult to align the schedules of multiple Value Streams to avoid both the delay before an anticipated delivery and the inventory waste of premature deployment.

One of the more unexplored areas of Responsive Deployment is its market impact. Nothing frustrates users like having their functionality changed out from under them, or to be forced to take hours to download and install the latest release. So while it’s touted as a good thing, in fact, the majority of the time it’s a bad policy to do Responsive Deployment. Instead, Responsive Deployment should be a set of mechanisms that apply selectively according to business mandates. In the spirit of Whack the Mole, a team should still always drop what it is doing to fix a product defect when it arises; whether one interrupts the client’s daily life to deliver that fix is a matter of policy and, as described above, dependent on the urgency of the fix and the relationship to the client. Responsive Deployment of features usually makes better sense when a team is serving a single customer that they can easily bring into the Community of Trust. In a high-performing team, Responsive Deployment of defect repairs should be standard operating procedure.

Responsive Deployment sucks with multiple development threads. If you’re continuously deploying software to multiple markets with multiple needs, sharing a common code base, the chances are high for unintentional contamination across them. Responsive Deployment is more likely to work in simple setups.

[1] Ryan Paul. “Exclusive: a behind-the-scenes look at Facebook release engineering.” In Ars Technica, http://arstechnica.com/business/2012/04/exclusive-a-behind-the-scenes-look-at-facebook-release-engineering/3/, 5 April 2012 (accessed 9 May 2017).

[2] Chuck Rossi. “Rapid release at massive scale.” In Facebook Code, https://code.fb.com/developer-tools/rapid-release-at-massive-scale/, 31 August 2017 (accessed 4 October 2018).

Picture credits: winterbee / Shutterstock.com.