Refined Product Backlog**

Alias: Backlog Refinement Meeting

Careful arrangement.

...you have a Product Backlog and the Scrum Team needs to look ahead in its planning.

✥ ✥ ✥

Agile enterprises must be poised to respond quickly to capitalize on opportunities to create value, and should avoid working—or planning—too far ahead. To respond quickly to a market opportunity, the team needs to understand the stakeholder needs, and all of the ducks must be lined up to enable work to proceed on the items at hand. The Product Owner must have communicated how Product Increment supports the vision. The Product Owner has time-ordered the deployment of Product Increments to meet peak market responsiveness, to create the highest value. When opportunity knocks, one wants to be ready for it. There may be a “last responsible moment” for market decisions; if it passes, the march of time and the resulting changing circumstances can take away your options or your ability to exercise them.

However, it’s also important to defer work that won’t generate value for a long time. Working prematurely on a long-term deliverable can have a high opportunity cost if it moves other viable product offerings out of their prime market window.

Further complicating this is the fact that some deliverables have hard dependencies on others. Some Product Backlog Item (PBI) may build on an earlier product increment; it’s important to plan ahead so the right foundations are in place when the opportunity for the big market win presents itself. This may imply that you deliver some PBI earlier than you otherwise would. Although an early release may generate lower value for that item, it may set up a Product Increment with even greater value.

As time goes on, the team learns more about the market and about dependencies. However, at the same time, people inside the organization make development and market decisions, or the market itself moves in directions that may demand or provoke changes to existing plans. Every new decision is a constraint that could limit future product and market possibilities. For example, one may lose the opportunity to be first in a market by deferring the introduction of some feature in that market. The team may order PBIs to gain the highest value given today’s understanding of the team and the market, but tomorrow may bring a window of opportunity that could create even higher value with a different backlog ordering. It is often difficult for the team to foresee these problems far in advance.

Therefore:

The Scrum Team (particularly the Product Owner and Development Team) should meet frequently to properly order the Product Backlog and to break down the most imminent large PBIs into smaller ones. The Development Team should maintain current estimates for the Product Backlog Items that it will eventually implement (Pigs Estimate).

The team should focus particularly on items near the top of the Product Backlog. Pay particular attention to dependencies between PBIs as well as to dependencies of PBIs on external market factors (e.g., holiday sales seasons) and to dates when resources may become available to make it possible to build the PBI (e.g., taking delivery of raw materials or enabling technology from a supplier). The Scrum Team should annotate PBIs on the backlog with estimates and value attributions.

PBIs near the top of the backlog (those for the next two to three Sprints) should be small enough so that no single one will require more work than 10 percent of the Sprint development effort. See Small Items.

✥ ✥ ✥

Refinement includes detailed requirements analysis; working toward a Granularity Gradient in the Product Backlog’s structure; bringing estimates up to date; reordering items to reflect dependencies between them or constraints related to the timing of a PBI release; and, in general, updating the Product Backlog to reflect any new information available to the team.

Keep in mind that the main purpose of bringing the team together to refine the backlog may be to get the team to start thinking about how to implement the envisioned features. Refinement is part of a learning process. Whether processing the envisioned need in their subconscious, or whether through informally discussing upcoming work over lunch, such advance mental processing of upcoming features fuels the breakthroughs that reduce visions to designs. Isaac Asimov proposed that creativity blossoms in the solitude away from collective activities and in the informal jovial exchanges outside structured professional settings ([1]). Steve Johnson points to “the slow hunchˮ as a major factor in creativity; backlog refinement and estimation plant the seeds that kindle this process in the minds of the team members. Seeing an item well ahead of its deployment gives time for this process to unfold (see ‟The Slow Hunch” in [2]).

One good first-cut ordering technique from the lean canon is the Design Structure Matrix [3] (see [4]). Make a matrix each of whose axes are labeled with the PBIs, putting an “X” in the cells where there is a dependency between items:

Then sort the matrix so that it is a lower-left diagonal matrix. That is, you pull the last responsible moments forward—you never defer them. Such was the original, pre-XP sense of the phrase “last responsible moment” ([5]). If it is a lower-left diagonal matrix, then every item will come after all of those on which it depends. Further, pulling these decisions forward leads the team to dive into the associated work, which will flush out emergent requirements and increase the level of insight into the problem at hand. The passage of time alone is no guarantee that any new insights will come along: the team must actively take an item into design and implementation to make emergent requirements appear.

Deal with hard external dates using common sense. If it is impossible to sort the matrix, then there is a circular dependency and the team should review the items with more scrutiny. In the worst case, the team may combine two mutually dependent items into a single, larger item.

With the fundamental dependency constraints in hand, use the remaining ordering freedom to optimize market alignment and value opportunities. Knowing the Development Team’s forecasts of relative costs and the anticipated value the item will generate at the time of delivery, you can adjust the ordering of items to optimize value. Just changing the order of a given set of items can often double the realized value.

Work as diligently as possible to bring each PBI (near the top of the backlog) in line with the Definition of Ready.

Temper the refinement with Granularity Gradient. Focus on refining the PBIs for the upcoming two or three Sprints and work on the rest if time permits—and if you have the information to make informed, responsible decisions.

In the early days of Scrum, this activity took place during Sprint Planning. As history progressed, teams started to spread the work between Sprint Planning and a weekly meeting sometimes called “The Wednesday Afternoon Meeting” or “The Product Backlog Refinement Meeting.” In contemporary practice it is common to spread the work even further in short, daily meetings. Each team can adjust the frequency and duration of these activities as necessary, but the total amount of face time that the Product Owner requires from the Developers should not in general be more than 10 percent of the working time. The Scrum Team should schedule all such activities in advance: it is not work the Product Owner may request on demand.

As in Sprint Planning, the team makes adjustments to the backlog not only to strive for the Greatest Value but out of sheer Product Pride.

For more on estimates, see Estimation Points.

Backlog Refinement is not a proper event in The Scrum Guide. [6] One argument that has been given for this irregularity is that there is nothing in the Scrum Guide that pertains beyond the current Sprint. However, that argument is mistaken, as The Scrum Guide says, ‟Product Backlog items that will occupy the Development Team for the upcoming Sprint are refined so that any one item can reasonably be ‘Done’ within the Sprint time-box,ˮ and ‟The result of the Sprint Review is a revised Product Backlog that defines the probable Product Backlog items for the next Sprint.ˮ [6] A more common interpretation is that the activity need not be held on cadence nor conducted as a ‟meeting,ˮ giving the Scrum Team the freedom to carry out the activity as they best see fit.

Pulling planning activities for future Sprints into the current Sprint is inspired in part by the overlapping Sprints of “type Bˮ production in the classic Harvard Business Review article by Takeuchi and Nonaka ([7]).

[1] Isaac Asimov. On Creativity. MIT Technology Review 118(1), Cambridge, MA: January / February 2015, pp 12-13.

[2] Steve Johnson. ‟The Slow Hunch.” In Where Good Ideas Come From. New York: Riverhead Books, 2011, Chapter III.

[3] —. “Design structure matrix.” Wikipedia, https://en.wikipedia.org/wiki/Design_structure_matrix, 17 February 2016 (accessed 2 November 2017).

[4] Steven D. Eppinger and Tyson R. Browning. Design Structure Matrix Methods and Applications. Cambridge, MA: MIT Press, 2012.

[5] Glenn Ballard. “Positive versus Negative Iteration in Design.” In Proceedings of the 8th Annual Conference on the International Group for Lean Construction (IGLC-8), Brighton, UK, 2000.

[6] Jeff Sutherland and Ken Schwaber. The Scrum Guide. http://scrumguides.org/scrum-guide.html, November 2017, referenced 8 June 2018.

[7] Hirotaka Takeuchi and Ikujiro Nonaka. The New New Product Development Game. Harvard Business Review 64(1), January 1986, pp. 137-146.

Picture credits: Shutterstock.com.