Agile Development Blog

This post will be split into two parts so that it, itself, will be half-assed with the suggestion of an additional half-ass to be delivered later :) And, in keeping with that spirit, the post mistakenly went live before it was ready for prime time. This time, I meant to push the ‘publish’ button…

Can you deploy Scrum to a test team?

Scrum is at its heart a product development process.  How can you leave the part of the organization – development – that actually makes product out of any Scrum deployment? Does it even make sense for other parts of the organization to be “doing Scrum” if development is somehow doing something else? Wouldn’t you be working towards what would be, at best, a half-assed deployment of Scrum?

Craig Larman and Bas Vodde in their wonderful book Practices for Scaling Lean & Agile Development certainly agree: “…a so-called test team Scrum is a contradiction in terms.” Ken Schwaber in The Enterprise and Scrum doesn’t seem to admit the possibility of deploying to functional groups – it’s projects he’s envisioning deploying to. For example, consider this advice for early goers of an enterprise-wide adoption of Scrum: “Establish preconditions that must be met before a project can use Scrum.”

The last 10 or so years of my career have been spent in big companies with very traditional structures: functional organizations with clear separation between development, test, usability, product management, etc; running projects that are very much plan-driven. Lots of agile practices that seem relatively straight-forward in other contexts aren’t in companies like this. Consider Schwaber’s notion of organizational deployment of Scrum again – this from the introduction of The Enterprise and Scrum: “This book is for those who want to use Scrum throughout their enterprise for product development.” It’s an awfully lucky convergence of thought and opportunity that would make such a deployment possible in large, traditionally organized companies. These sets of wholly distinct sub-organizations need to be both willing and able at essentially the same time. You might get a chance like that, but I wouldn’t hold your breath waiting for it.

You can start to see why that opportunity would be rare when you look at it from their perspective.  In taking a project-by-project focus in deploying Scrum to organizations like these – and assuming you’re holding firm to deploying every part of Scrum straight away – you’re essentially asking them to:

• Reconfigure their teams
• Change how they define and manage product scope
• Empower a single person to make scope decisions on each project
• Change how they plan their work
• Change how they approach their work in areas like development and testing
• And so on …

The point is that, even if limited to the context of a small set of pilot projects, they have to do all of this stuff first before they can realize the benefits of Scrum.

To me, this is exactly the same argument that we, as agilists, are very much accustomed to facing from development teams:  “We can do that feature but first we need to re-engineer the infrastructure to support it, which will take six months.”  We encourage teams making that argument to find ways to do just a bit of the refactoring to allow just a bit of the value to become realizable, rather than predicating all of the value on all of the refactoring.  Why can’t we make a similar argument in support of deploying Scrum to just a part of the organization?

What would Voltaire say?

One of my favorite lines – frequently quoted in optimization discussions but applicable equally well here – comes from Voltaire:  “Le mieux est l’ennemi du bien” (the best is the enemy of the good). “Best” is hard to define in any complex system like a large company but “good” seems a little more tractable. We should not let an inability to approach some notion of perfection prevent us from getting better.

A colleague was presented with this exact scenario a while back.  Representatives from a test group came to him asking if he would work with them to try Scrum.  He and I spent some time talking it over.

Things like product owner, product backlog and potentially shippable product increment looked like a tough fit for a test team, to be sure. But still, we thought of goals like “verify feature X” where the challenge to the team is to find a way to work together to get that done within the time-box of the Sprint. That might be a liberating shift in thinking after heavy doses of planning of the form: “We have a bazillion manual tests to execute. At 5 per hour, that’s bazillion/5 staff hours. With 20 FTE, that’s a bazillion/(20*5*40) weeks of testing.” Looking ahead to subsequent Sprints, we envisioned helping the test team bring some of their development partners into their Scrums – perhaps through broadening the notion of the verification goals to include “hardening” – finding and fixing bugs as a cross-functional team. And from there to the odd small feature, slowly working our way towards aligning the work of the test and development organizations in cross-functional Scrums. Even with such an odd scope of initial deployment, we could see potential benefits, including improved productivity through the iterate and reflect cycle, better planning and estimation and higher morale.  Not surprising, these are the same benefits we would suggest lay before any team looking to try Scrum.

Isn’t this the good that Voltaire would caution us against passing on?

Larman and Vodde have some great advice about how to go about ever more closely approaching the “potentially shippable product increment” goal of the Sprint through expanding the Definition of Done (DoD below):

“In general, these are the ways of expanding the DoD:

• Automate – for example, performance testing is automated
• Expand team cross-functionality – for example, a person with technical-writing skills joins the team”

That latter idea suggests a path to improvement that starts in development and then spreads over time to the remaining functions.  If we view Scrum deployment as being something we do in the context of projects and products, this is both natural and reasonable.  But, if we view deploying Scrum as something we do in the context of teams of people or if we view it simply as “transforming the world of work,” then why would we believe we have to start with any particular set of people?  Why not start with testing and grow our way towards development?

Would that be a half-assed approach to deploying Scrum?  Perhaps, but like Richard Dawkins’ half a wing or half an eye, maybe half an ass may prove a more useful incremental improvement than may be apparent at first glance.

So, can you deploy Scrum to a test team?

Sure, why not?

What do you think?

About the Author: Ed Willis has been a ScrumMaster, Product Owner, Scrum coach and trainer.  He is currently a developer in the telecommunications industry.

Here’s something obvious I’ve learned the hard way: delivering “potentially shippable product increments” each and every Sprint isn’t easy.  Essentially you’re trying to take all the disparate activities that occur throughout the waterfall process, focus them on just the little product increment’s functionality and then jam them into a teensy little Sprint.  Hard to do and definitely pretty unlikely to get accomplished right out of the chute. The authors of “Practices for Scaling Lean & Agile Development”, Craig Larman and Bas Vodde, forgive you in advance for not getting this done straight away. In fact they suggest that this is more a goal the team will approach over time than a rule they put in place on day one.

The authors propose something very simple but very insightful:

• Sketch out all the things you need to do to get your product out the door.
• Define “done” as that subset of those the team is currently capable of performing every Sprint.
• Use your retrospectives to challenge the team to bring more and more of the “undone” work into each Sprint.

This has already changed the way I think about retrospectives. For other nuggets from Larman and Vodde’s book, read on.

Overall a Must-Read for Agile Development Leaders

I was blown away by “Scaling Lean & Agile Development”, as you can see from my bullish review on O’Reilly.  Some time has passed since then but I still feel that it’s one of the most important development books I’ve read.  That book alluded to the companion volume, “Practices for Scaling Lean & Agile Development”, and as you can imagine I awaited its publication eagerly.  It came out in February – I’ve worked my way through it now.   It’s most definitely a worthy successor.

The first book presents theoretical and philosophical underpinnings for agile and lean development. The second book presents a survey of practices relevant to all aspects of the process of developing software at scale, presented by two guys who bring a wealth of knowledge and experience to the table.

Continual Investment in Requirements

Larman and Vodde present practices relevant to a continual investment in requirements throughout the product development process. This is done both up-front, in seeding the product backlog, and iteratively, in refining requirements to support upcoming Sprints.

I love the emphasis they place on whole-team involvement in the initial Product Backlog development effort – even for projects with large teams.  Too often I’ve seen this be the provenance of Product Owners, working in near isolation, which does little to get the project off on a good footing.  The first the team sees of the requirements is the Product Backlog itself, having been involved in none of the  discussions and thought that went into it.

The notion of requirements areas, which was introduced in the first book, is leveraged here to help parallelize the initial requirements development work.  Concurrent sizing and value estimation workshops keep the requirements work rooted in reality.  They spend some time on the problem of bootstrapping a consistent sizing process in a large scale team. The use of cross-team estimation sessions result in the development of a canonical set of sized stories used as a basis for subsequent sizing at the team level.

In Favor of Forward-Looking Requirements Refinement

The discussion of forward-looking requirements refinement really resonated with me. I’ve found in my own practice that peeking ahead at the Product Backlog items upcoming in the next Sprint and then spending time together working through them to understand what they really mean – before we get into the Sprint planning session – goes a long way towards supporting a predictable iterative process.

It also puts the requirements elaboration just barely outside the Sprint that the work is planned for.  This separates coming to an understanding about what we want from the stress of figuring out how to fit it into a Sprint, which makes for more open and enjoyable requirements development.  The authors suggest the use of examples and Acceptance Test Driven Development (ATTD) to more precisely capture the intended behaviors of the stories in a way that realizes both conversation and confirmation of the “three Cs” . Smart.

The authors show how traditional approaches to project scoping and commitment (where separate product management and development organizations act essentially as competitors) are very much analogous to the contract negotiation that the Agile Manifesto cautions us against.  I’m surprised to admit this never occurred to me – I always read that part of the manifesto pretty literally. But it’s inarguably true that the wrangling between these two groups over project scope and timelines that happens in many organizations is a form of contract negotiation, and the waste it drives can be startling.

The arm wrestling of the product management “market ask” captured in a Market Requirements Document followed by the team’s “development response” carefully defined in a Product Requirements Document is a case in point.  Weeks and months can go by during this ritual.  What are the development teams doing during this time?  Not at lot, in my experience.  Does development really “win” if they manage to push out the end date or reduce scope?  Does product management really “win” if they manage to squeeze in extra features or pull in the release date?  I’ve seen how the empowerment of the Product Owner and the establishment of the Product Backlog as the single high-level requirements and release scoping artifact helped to prevent some of these painful dysfunctions but I’ve never thought of them in terms of an imbalance of contract negotiation over customer collaboration.

Creating “Desire Lines” in Design

Larman and Vodde strongly advocate wikis as the preferred basis for the technical documentation the team develops.  They suggest a page for the overall product and pages for each Sprint. That worked for my teams as well, although there was always a certain (healthy) tension regarding what was appropriate for Sprint documentation and what should be living documentation at the product level.

They present the compelling idea of “desire lines”, which, in landscaping, refer to paths that develop naturally, as people use a given space.  Rather than guessing how people would use the area, they are observed using it and then the landscaping is designed around their actual usage.  Similarly in design, rather than trying to guess what the needs are, let them emerge and then refactor to support the emerging design tensions.  A lovely analogy, I thought, and one that will stay with me.

They suggest both a priori design in collaborative design workshops and design after the fact in System Architecture Documentation (SAD) workshops.  I like the acceptance that both approaches are going to be useful. The former stresses the need for team contribution to the design as the Product Backlog items are being developed, while the latter recognizes that team needs for technical documentation will emerge over time and so setting time aside to fine tune design documentation is both warranted and healthy.

The authors also address the question of whether and when to rewrite code – like Joel Spolsky, I personally favor refactoring to improve legacy code rather then re-engineering.  The authors present a compelling and complementary argument for refactoring and incremental improvement based on the value of instilling a notion of continual (rather than punctuated) improvement in the development teams.  They stress that the real problem isn’t the legacy code you’ve got but rather the legacy code you continue to write.  Encouraging the team to have a mindset of each check-in leaving the code base better than it was before – fixing the broken windows and taking out the trash – is a better solution to the problem of poor code quality than is carving off large sections of time for exclusively improvement-oriented work.

Acceptance Test Driven Development

The book’s worth it just for this material alone.  Larman and Vodde present a wonderful analysis of ATDD and characterize its place in the context of Scrum, including tying it into iterative requirements refinement, the Definition of Done and the Sprint Review.  I’d seen teams go in this direction driven largely by the desire to better engage testers in their teams throughout the Sprint and avoid “scrummerfall.” The authors take this further to show how ATDD, in which acceptance tests are defined and automated in advance of the development of the code that will pass those tests, does for teams in an iteration what Test Driven Development (TDD) does for individual developers in ten minutes.

They stress the need for the test automation to be a distributed responsibility shared by the whole team rather than one assigned to a specialist team.  I couldn’t agree more.  I’ve seen many attempts to establish test automation through the creation of a group apart from development and I can’t say that I’d call any of them particularly successful – at least not in the broad and encompassing sense that Larman and Vodde are envisioning in this book.

Continuous Integration at Scale

I was particularly glad to see treatment of Continuous Integration (CI) at scale. I’ve seen groups start with vanilla CI as it is described in the Extreme Programming literature and do well with it initially but then fail as their groups grew larger.

Essentially, Larman and Vodde propose a nested set of continuous integration builds, each larger one subsuming sets of smaller ones within it and each build defined for a particular concern.  Examples of these concerns include the verification of specific components and subsystems but also particular kinds of testing – including things like performance and stability testing.  One key aspect of this approach is that, at each level, you’re trading off the immediacy of feedback to the developer against the likelihood of the developer’s submission affecting quality at that level and the expense of the tests.

The Elusive Definition of Done

As noted earlier, Larman and Vodde,  suggest defining “done” as that subset of the work needed to release the product the team is currently capable of performing each and every Sprint – then use your retrospectives to challenge the team to bring more and more of the “undone” work into each Sprint.  The authors point out that there are really only two ways to extend the Definition of Done:  increase the cross-functionality of the team or increase the degree of automation the team uses.

In the meantime, however, while the Definition of Done leaves some work undone, Larman and Vodde suggest meeting that problem honestly, by doing things like defining an undone work team and pushing undone work onto the Product Backlog.  By being explicit about where the team currently stands against the goal of delivering potentially shippable product increments, this undone work can be better managed and the team’s insights can be directed to the problem of expanding that definition of done to get them closer to the goal.

For the Bookshelf of any Agile Team Member

The book isn’t without its faults.  It’s certainly long enough and some sections can be tedious (there’s a fifteen or so page section where different scenarios for story splitting are laboriously addressed – I get that this common complaint about stories needs to be put to rest but this feels like killing me with kindness :)).  The book’s organization lends itself more to use as a reference than as something you’d read cover to cover.   There are many repetitive sections that allow each chapter to stand on its own but are a bit hard to get through when you read it straight through.  But these are really just quibbles.  In all, “Practices for Scaling Lean  & Agile Development” is, just as its companion volume before it was, a tremendous book that belongs on the bookshelf of any agile coach, manager or team member.

I’d like to thank Anne Greenhaw and Selaine Henriksen for their help in developing this post.  Thanks also to Ryan Martens for inviting me to post here.

About the Author: Ed Willis has been a ScrumMaster, Product Owner, Scrum coach and trainer.  He is currently a developer in the telecommunications industry.

Managing the right goals in software development can have a major impact on your team’s success. You get what you measure. In this regard, the goals you choose can be a defining factor in whether your organization’s Agile adoption thrives or dives. So, what are your choices?

A story about goal-setting

Imagine you are a runner. You have a running coach defining targets for you. Your coach chooses miles and seconds in order to measure you. These metrics turn out to be useful; they are universally understood by you, your coach and your competitors. That’s a good thing. A target goal exchange around these metrics might look like the following:

Coach: “I need you to run a 2-minute mile.”

You: “Uh, I’m not sure I can do that. In fact I don’t think it has ever been done before by any land animal other than a wildebeest, a pronghorn antelope or a cheetah. And, I’m not a wildebeest.”

Coach: “You’re not listening. I already told Sports Illustrated you will. Don’t make me look bad.”

You: “Seriously I’m pretty sure that’s just not humanly possible. I’m no Usain Bolt and I’m not sure even he could sustain a 2-minute mile.”

Coach: “Hey, I set the goals. Run that mile in 2 minutes. Just hunker down and work harder. 2-minutes is the target goal. If you don’t meet it, I’m kicking you off the team.”

You: (muttering under your breath to self, “I can’t do it, you aren’t listening to me, and you are a moron. Why do I even bother to open my mouth?”)

Why goals matter

Get the idea? I’ve been reading John Seddon’s Freedom from Command and Control: Rethinking Management for Lean Service. My colleague Alan Atlas recommended the book. Then my manager Ryan Martens became very interested which led to a book appearing on my desk. Soon after, Karl Scotland mentioned it in an informal conversation about Lean and Kanban. Time to pop it to the top of my reading stack!

Seddon sets a context about Purpose driving Measures that then drive Methods. The combination of purpose and measures start to look like goals. And Seddon breaks goals into two types: Target Goals and Capability Goals. While John’s book is not specifically about software, I could see how we use and abuse goals in our software world. So I decided to delve deeper.

Target goals are arbitrary and doom-laden

Target goals are set to measure hopeful results, often an arbitrary number. Think about our runner story and the 2-minute mile. That target goal was  set by someone else. A target goal typically does not account for your strengths, capabilities , availability and skills. Target goals don’t serve you. They don’t serve organizations either. And yet we cling to them. Consider if, as a developer, I hear the following from my manager:

“Jean! You missed the release deadline we gave you. We told marketing and sales who then told our customers that dev would deliver this specific functionality on this specific date. You’re ruining everything. We’re doomed I tell you, doomed! And all because YOU didn’t hit YOUR deadline.”

Uh, it wasn’t my deadline. And yet, missing that goal is now rippling negatively through the organization. This is a lose-lose situation all due to expectations hinged to an arbitrary target goal.

So why do target goals exist?

In many hierarchical, command-and-control software organizations, target goals are how deadlines are set. As Seddon points out, use of ineffective target goals usually arises due to an impossible purpose coupled with non-value add methods. The measures (or goals) are meant to drive teams to the impossible purpose. A command-and-control organization believes that goals must be set top down. The negative impact of these goals is inextricably woven into unrealistic expectations, death marches, and non-value add methods for controlling the work toward the goals.

Enter the capability goal

Let’s replay the runner scenario again.

Coach: “There is a race coming up, a half marathon. That means we need to start training now. What is your time currently?”

You: “I haven’t been really running lately. Rather than guess my times, I’ll start running now. Do you want to know in terms of how long it takes me to run a mile, or how far I can run in a specific timebox?”

Coach: “Let’s start with how you finish a mile. What is your typical way of getting going?”

You: “First, I do a walk/run combination. Then I can give you my actual mile capability in a week.”

Coach: “Okay, let’s set up a plan and together set some sort of goals based on your capabilities. That will help me guide you and figure out when we are faltering. I know you can do this. I’m going to work with you starting right now.”

You: (muttering under your breath, “This is amazing; yea me! I am going to finally put my passion for running to work, keep improving, and run my first half marathon. I love my coach!)

Capability goals are based on real data

Together, the runner and the coach watch the capabilities of the runner to define goals. The purpose of the goal is clear. Measures are set up that usefully help determine the runner’s progress. The method (or running regimen) for attaining the goal is then applied. Simple.

We use this same approach in Agile teams. A team lead or project manager does not set an arbitrary target goal. Rather, the Product Owner sets an overall purpose/vision. The Agile lead, often referred to as the ScrumMaster, works with the team to assess what goals can be reached based on a team’s capabilities and availability. From iteration to iteration, the team provides feedback on what it is able to complete based on its capabilities. The Product Owner absorbs this information and sets expectations outside the team. The process of continuous improvement continues. The team declares, “Based on what we tend to complete, and given the purpose from the Product Owner, we are going to improve our methods in the following ways.”

When good capability goals go bad

There’s a Gary Larsen “Far Side” cartoon of an open refrigerator with a bowl of glop inside holding a gun. The caption? “When good potato salad goes bad”.

Unfortunately, I’ve seen good capability goals go bad in a number of organizations. Listen to this conversation:

Program Director: “I see that our Agile teams are all now using these things called story points to track their work.”

Me: “Yes, it’s working well. Each team has a good sense of what it is capable of delivering iteration over iteration. That helps me figure out how to help remove impediments, where bottlenecks are, and where any team may need us to better resource them. It’s great!”

Program Director: “Hmmmm.”

Me: (muttering under my breath: “That ‘Hmmmm’ just doesn’t sound good. I think I’m about to get sick.”)

Program Director: “So, what’s the highest story point commitment a team has made?”

Me: “27.”

Program Director: “Great!  I’m setting 27 points as the target goal for all teams to commit to for each iteration. Every point will represent 3 hours of work. This precision will help me calculate exactly what we’ll deliver when. Plus, I’ll know which teams are really working and which ones are slackers.”

Me: “Hmmmm.”

Program Manager: “So, let’s get out there and commit! This Agile stuff is great!”

Me: (muttering under my breath, “We’re doomed, I tell you doomed!”)

What just happened must not be allowed

As you roll out your organizational Agile adoption, you’ll encounter many challenges. Pay attention to the usefulness of capability goals. Insist on using them.  Non-Agile organizations won’t like this. At times, you may feel like Sisyphus pushing a rock up hill for all eternity. Be strong. Don’t allow Agile measurement abuse. Create team and organizational trust.  Remove target goals and embrace the incredible value of capability goals. And, don’t allow any backslide. Don’t let your capability goals devolve slowly into target goals in sheep’s clothing.

You: (muttering under your breath, “You know? This MAY just work. Capability goals versus target goals. For the sake of my team, our organization, and the value we deliver to our customers, I’m going to do it!”)

Brad Murphy, CEO of Gear Stream, gave a keynote presentation at last week’s Agile Success Tour. Brad’s vision includes helping organizations transition from project-driven governance to Lean, continuous flow, product line execution. We sat down with Brad to ask him 5 questions about Agile.

1. What’s the most valuable thing you’ve learned by helping organizations adopt Agile practices?

The dynamics surrounding the adoption of Agile practices are changing in ways that are very different from even two or three years ago. In parallel with implementing Agile practices and establishing effective Agile teams, we’re helping organizations connect Agile philosophies and values to vital stakeholder groups. We’ve recognized that actively involving all stakeholders early in the process is key to ensuring that core Agile teams are able to achieve expected results. Gaining the support of all collaborators in the organization dramatically helps core Agile teams successfully transform and deliver outcomes. Involving the following stakeholders is vital, including:

• The core Agile team
• Tooling, Infrastructure and Release Management – the architecture side of the equation.
• Governance and PMO groups – more than half the time, this group builds the roles of the project managers, Agile teams and the Scrum Master. They consider how Agile teams work off backlogs. There’s huge potential for misalignment here.
• Product Management – since this is frequently a group that doesn’t exist, the responsibilities here aren’t well defined. Loosely, I’ll describe this as a group of individuals responsible for defining business value. This group has to be aligned with the way Agile works.
• Executive Management – all of the focus on Agile change with the groups listed above must be explicitly aligned with the strategy and values articulated by senior executives. Too often, Agile is pursued based on “generic” benefits like speed, responsiveness, quality… while these are “good,” they’re not explicit enough.  We must challenge executives to express exactly how they will measure/recognize their own corporate goals and then map these back to the things we choose to emphasize in an Agile transformation.

Responsiveness would be #1. Having the ability to course-correct and change directions quickly in light of new mandates and directions. A close #2 is innovation.  A distant #3 is productivity.

Gear Stream’s clients are realizing compelling organizational realignment by refocusing the entire value chain on external customers, as opposed to the often dysfunctional inward focus of many organizations.  By aligning the entire value chain on external customers, our clients are able to more predictably influence customer satisfaction and marketshare objectives.

I’ll give the advice I’m most passionate about. Agile teams need to challenge the rest of the organization to rethink and reconsider how work happens up and down-stream to them in order to fully exploit what they’re capable of achieving. It’s really about how Agile teams can actively promote, influence and change an organization beyond just what they do as a team. They really can – and do – influence and engage the rest of the organization. The failure pattern of Agile is almost always the same: we’ve built an Agile team, but failed to build the adjacent stakeholder teams in the process – then, the Agile team gets frustrated and the organization loses momentum and enthusiasm for Agile. Agile teams must have the ability to influence stakeholder teams, and ultimately, the baseline infrastructure of the entire organization.

5. How do you define Agile success? (Can you do it in 113 characters?)

Building, delivering & sustaining outcomes that customers r continually thrilled by, driving profit & value 4 all

I would like to welcome Ed Willis to our blog, as a guest blogger.  I spent a couple of days with Ed and his team of internal coaches this Winter and I am glad to have him share some of their lessons from the field of large distributed Agile – Ryan

Question: What are we, as Scrum team members, committing to in a sprint?

The higher-level Sprint Goal or the more detailed set of selected Product Backlog? If you are a customer of the team, what is the team telling you it will do?

I recall being pretty confused about what the team was committing to in Sprint planning after reading Agile Software Development with Scrum, by Ken Schwaber and Mike Beedle, for the first time. There seemed to be some ambiguity around whether the Scrum Team commits to the set of selected Product Backlog items or to the Sprint Goal, which is a higher level and less precise concept. A couple of quotes from that book will help illuminate the situation:

“A team commits to achieving a Sprint Goal.”

“The Scrum Team commits to turn a selected set of Product Backlog into a working product.”

And this from Schwaber’s The Enterprise and Scrum: “… the team selects as much Product Backlog as it believes that it can transform into a completed increment of potentially shippable product functionality by the end of the Sprint. The team commits to the Product Owner to do its best to complete that amount of functionality.”

From the first book again, “The Sprint Goal is an objective that will be met through the implementation of the Product Backlog … The reason for having a Sprint Goal is to give the team some wiggle room regarding the functionality … If the work turns out to be harder than the team had expected, then the team might only partially implement the functionality. “

Note that that last quote appears nearly verbatim in a recent (Feb 2010) version of the Scrum Guide published on scrum.org.

Between the current Scrum Guide and 2001’s “Agile Software Development with Scrum”, Schwaber’s Agile Project Management with Scrum presents an overview of Scrum in its first chapter which does not discuss the Sprint Goal at all – similarly, “The Enterprise and Scrum” provides a summary of Scrum in an appendix that does not talk about the Sprint Goal.

So what are we, as Scrum team members, committing to: the higher-level Sprint Goal or to the more specific set of selected Product Backlog? And why am I so focused on Sprint-level commitments? I focus on commitments because my stakeholders and customers value them. Their plans are in part based on my team’s delivery and so being able to make and meet short-term commitments makes us a better partner to them – “don’t trouble your customer”.

Can You Commit to an Incomplete Sprint?

“Agile Software Development with Scrum” states, “Sometimes only a partial Sprint Backlog can be created … define the initial investigation, design, and architecture work in as much detail as possible, and leave reminders for work that will probably have to be done once the investigation or design has been completed.”

Both that quote and the earlier one on the purpose of the Sprint Goal suggest that the Scrum canon allows for both incomplete Sprint planning and a mechanism to de-scope the Sprint if the team falls behind. If you find yourself in a situation similar to mine (and I suspect many, many of us find ourselves in this situation) where your customers and stakeholders really do value the ability to make and meet high-level commitments in terms of working software that will be delivered at Sprint’s end, then I think that finding a way to do this more regularly is a path you can and should pursue. And it’s one my colleagues and I have embarked on wholeheartedly. All you really need to do to become a better partner in this regard is advance your Sprint planning practices and adapt the strategies you use when you discover you’ve fallen behind.

Reliably Committing to the Selected Product Backlog – One Group’s Perspective

Being able to reliably deliver against Product Backlog commitments isn’t hard conceptually – it starts with doing a better job in Sprint planning. There are probably many ways to do this but I thought I’d share what we’ve done to accomplish this.

Capacity Planning

Our Sprint planning starts with a detailed view into capacity. Is anyone on the team going to be away on training or vacation during the Sprint? Will the remainder of their time be focused on the Sprint? If not, how much do time they think they’ll have to spend in the Sprint? Knowing the total capacity of the team in the coming Sprint is the foundation of a solid Sprint plan.

Ideal Time Estimation

To estimate our Sprint Backlog tasks, we use “ideal time” as it is described in Planning Extreme Programming by Beck and Fowler: “Ideal time is time without interruption where you can concentrate on your work and feel fully productive.” Ideal time on any given task is in essence the answer to the question “how long would this task take you in a perfect world?” Ideal time is then converted into real time in a very similar manner to how story points are converted into expected duration for stories – you measure the velocity of the team. For example, if ideal time velocity of a team is measured at 0.66, then two ideal days of work has been seen in the past to be about three calendar days.

That use of past experience is critical because it rolls up estimation error, overhead, interruptions et al into a single factor used to improve the accuracy of the task estimates while only requiring of the people doing the estimates that they do so consistently. We’ve found it useful. Conceptually I also like it because, along with the use of story points and their corresponding velocity, it increases Scrum’s fractal, self-similar nature.

Assigning Tasks

Typically we assign all the tasks in the Sprint Backlog during the planning meeting. There’s probably enough divergence of thought on this point to warrant its own post, but to stick to the matter at hand, we do this for reasons relevant to the goal of producing more reliable Sprint plans. One major motivation is the fact that estimates provided by the person who will actually do the work are demonstrably more accurate. The second major motivation is our observation that, when trying to achieve greater work sharing and improve the flow of value to the customer by avoiding bottlenecking through specialists, we do a better job of supporting people who are branching out into unfamiliar territory if we know they’re going to do so a priori. Specifically, we can build time into our Sprints for the people who are more knowledgeable in a given area to mentor or review the work of those who are working in areas less familiar to them.

Spikes

We make frequent use of spikes to allow us to develop a better or more complete understanding of what a given story will entail thus giving us the insight needed to build more complete Sprint plans.

Reviewing the Sprint Plan

We review our Sprint plans to look for mistakes we commonly make. We’ve gone as far as enumerating the mistakes we make most often to help us look specifically for those as we do Sprint planning. Most of these are different flavors of not applying our “definition of done” and thus ending up with incomplete plans. If your goal is to develop challenging plans but plans that you’re likely to deliver on, then trying hard to ensure your plans are complete is important.

Note that none of the above should be intended to suggest that we value immutable Sprint Backlogs or task assignments. We can and do frequently grow and shrink the Sprint Backlog as the team uncovers more information about what’s needed to deliver the committed Product Backlog. Similarly, we shuffle (and re-estimate) tasks regularly. What we don’t do is allow significant parts of the work to go unplanned or unassigned at the point we decide what we’re committing to deliver.

Beyond Yes or No

Beyond improved Sprint planning, the second critical aspect for us in delivering committed Product Backlog in a Sprint is how we deal with falling behind. I frequently say in delivering Scrum training that if someone comes to you asking if a bunch of features can get done given fixed resourcing by a given date, that the best of all possible answers is “yes”. The worst of all possible answers is “no” – but not because you’re delivering bad news but rather because saying “no” reinforces the fallacy that that decision is really binary in nature and ignores the great value to be gained by digging into the details to see what’s actually possible.

Similarly when Scrum teams fall behind, they frequently see their choices as being to either: stick with the plan or remove lower priority Product Backlog in consultation with the Product Owner. Those are reasonable choices but I much prefer a third option: work very closely with the Product Owner to find an easier way to deliver the committed Product Backlog items in the time remaining – essentially, use the combined talents and insights of the Product Owner and the team to find simpler, cheaper ways to get the selected Product Backlog done. Honestly, it can feel like magic when it happens.

We once had a Sprint on a tool development project that had fallen far behind – it appeared that no amount of plan scrubbing or task shuffling would allow us to deliver the committed Product Backlog by Sprint’s end. We met with the Product Owner with the intention of choosing Product Backlog to remove from the Sprint. In the course of conversation though, the Product Owner got a better feel for the approach the team was taking in developing one specific item – the item that was driving most of the over-run. Collectively, they realized that the approach was fancier than was really required and in short order sketched out an alternative. We immediately revised our Sprint Backlog to remove the old approach and built a plan for the new one – in about an hour, we went from hopelessly behind with no chance of delivering the complete set of Product Backlog to being close to on schedule. I think having a strong team desire to deliver the whole of the selected Product Backlog was key to making this happen.

Anecdotally, I can say that my colleagues and I worked hard to improve our Sprint planning and replanning practices and pretty quickly found ourselves doing much better in meeting our Product Backlog commitments. For example, during one ten month stretch, I counted 93% of Product Backlog items delivered on time in the Sprints in which they were planned. This isn’t the only thing that matters but it made a big difference to us and our stakeholders: it gave us confidence in the team.

Conclusions

• Use the Sprint Goal to capture over-arching but incremental project goals if this concept is meaningful to you and if goals like these are more meaningful commitments than would be the selected Product Backlog – but definitely don’t use the Sprint Goal as a crutch to support poor Sprint planning practices.
• Don’t let the Scrum canon hinder your team’s thinking when tuning your process to better satisfy your business needs – the canon wasn’t created to limit your thinking.
• And above all, don’t believe that your team won’t be able to reliably commit to the selected Product Backlog Sprint over Sprint – you almost certainly can and, if your team and your stakeholders value this, then by all means pursue improvements in this area.

I’d like to thank Anne Greenhaw, Selaine Henriksen and Ryan Martens for their help in developing this post.  Thanks to Ryan also for inviting me to post here.

About the Author: This is the first guest blog post of Ed Willis. Ed was a soldier, a cab driver, a security guard, a tree planter, an auto glass installer and shop manager, an employee of several bookstores, a house painter, a dish-washer, and a foot messenger, among at least a few other things before buying his first computer. Since then he has worked as a research assistant at Queen’s University, driven a data mining company into the ground and ridden Nortel Networks nearly all the way down. He currently works for company in telecommunications. Born in NY, NY, he now lives in Ottawa, Ontario.

Today we announced Rally’s acquisition of AgileZen, a visual project collaboration tool that manages work using the Lean concept of Kanban. AgileZen is a simple, elegant project collaboration tool that supports software development by providing a Web-based Kanban board.

Our definition of Kanban

If you aren’t yet familiar with Kanban, there are lots of great resources out there. The simplest description we could come up with for Kanban and Scrum is in our press release, but we welcome your thoughts and additions:

Kanban literally means “sign board,” and in Lean it is the signaling tool for visualizing and tracking work as it flows through various stages of a process. A Kanban board does this by exposing bottlenecks, queues and waste in a process so that teams can deliver high quality, high value work. Both Scrum and Kanban methods focus on early value delivery, and both provide transparency into the work in progress.  But Kanban can operate with a different planning and delivery cadence than Scrum and emphasizes different metrics.

What does this mean for Rally and the industry?

We believe that Kanban is a simple but natural extension of Agile software development. It will invite more teams into Agile and provide more runway for mature teams. But most importantly, it will help us extend Agile beyond development teams to create an Agile business. The AgileZen team has been effective in all of these areas. We are in heavy learning mode and, at least in our view, the entire industry is still figuring out how Scrum and Kanban work together and which methodology is better fit for various projects.

Welcome Nate and Niki Kohari!

Nate and Niki Kohari, co-founders of AgileZen, have built the best Web-based Kanban board out there, and we have the utmost respect for their product, company and brand. We are incredibly excited for them to join our North Carolina office.

What does this mean for AgileZen?

First, you should read Niki’s blog post. Not much has changed for AgileZen users. Together, we’ll continue to support the AgileZen solution as a low-cost Kanban-focused project collaboration tool, and users can access support as they always have.  If you aren’t familiar with the AgileZen product yet,  check out their free product.

Join us at the Lean SSC Conference in Atlanta next week!

The Rally and AgileZen teams will present our products and coaching services at the Lean Software & Systems Conference 2010 in Atlanta April 21-23. Ryan is also speaking on Plan-Do-Check-Act. We look forward to seeing you there!

About the Authors: Ryan Martens is a goat cheese maker,  founding board member of the Entrepreneurs Foundation of Colorado, and Founder and CTO at Rally Software Development. Jean Tabaka is a wine enthusiast, author and Agile Fellow at Rally Software Development. Subscribe today to get free updates by email or RSS.

Last summer, Rally started a video series called “Chalk Talks“.

I was fortunate enough to have filmed several “Chalk Talk” videos about some of the basics of Agile software development (The Agile Manifesto, Scrum Basics, Iteration Demo and Review Meeting, and other topics).

Since then, Rally’s expert team of Agile Coaches have joined the party and recorded additional Chalk Talks, again on some great basic Agile topics: Ronica Roth on User Stories, John Martin on Story Points, and Rachel Weston on Release Planning in Agile just to name a few.

We also tapped into the wisdom of some of our other Agile Coaches: Julie Chickering, Mark Kilby, and Ken Clyne.

Our Rally Chalk Talks are informal videos, typically 3 – 5 minutes long, intended to provide quick, easy introductions to Agile topics.

Filmed in a short, tutorial format, these videos are great to share with your team as they are getting up to speed on Agile.

To get a feel for our latest work, here’s Rally Agile Coach Ronica Roth in her great Chalk Talk on User Stories. ( during which you can find out why a dog would want his own laptop :D )

Be sure to check out our entire catalog of Chalk Talks and subscribe to our YouTube channel if you’d like to be notified when we publish new videos. We already have two more Chalk Talks queued up: “Kanban and Scrum” and “Agile and Lean”.

As you look through our current catalog of talks, be sure to let us know what other topics you’d like us to cover in future talks.

About the Author: Jean Tabaka is a wine enthusiast, author and Agile Fellow at Rally Software Development. Subscribe today to get free updates by email or RSS.

For 2010, lets find ways to focus on teaching our craft and growing the world of skilled software development professionals instead of trying to figure out who is “right.”

I believe much of the “Escalation” that Jean is seeing was correctly titled by Regina Mullen as a battle to be “right.” (see and read Escalation is Killing Agile – Can We Please Stop It? and Escalation is Killing our Healthy Conflict in Agile). That behavior focuses on carving up the pie instead of growing the pie.   There has been so much added to the field of software development methods, tools and techniques from the guiding ideas of Agile.  Now is not the time to stop and eat.

For me, 2010 is about continuing to grow the Agile software development pie’s reach and innovations.

I believe one of the key fixes to the problem of escalation can be found through increased professionalism and certification in Agile. By raising the bar through “difficult and skills-based certification,” as Brian Marick and the board at the Agile Alliance described, we can advance the Agile discourse through :

• a defined a bar that is deep in skill, knowledge and practice
• a significant enough bar to engage College and University study and examination
• research and curriculum that explore the tough questions in a scientific method
• development of more flexible or “T” shaped individuals that can see and work beyond silo roles.

With this back-drop, I am motivated by the notion of creating a  A Community of Thinkers,:

I am a member of a community of thinkers and I believe that communities exist as homes for professionals to learn, teach, and reflect on their work.

A Community of Thinkers creates more leadership in our profession. I see the expanding certification efforts in 2010 as great steps in these directions:

• Lean Software and System Professional – new certification emerging
• Agile Developer Certification – that is also emerging
• Certified Scrum Practitioner – especially as it intertwines with the PMI’s Agile Community of Practice

I encourage everyone in our community to figure out how to put energy toward one or more of these efforts.  The benefit of actively learning, teaching and reflecting on our work should lead us all to expanding civil dialogue that works to understand all points of view and keep expanding our thinking.  Thus broader education and difficult certification helps create a “Community of Thinkers.”  And, a Community of Thinkers will create a virtuous cycle of win/win and thus a larger pie for all.

That is my hope for 2010 in our profession.

About the Author: Ryan Martens is a happy father,  founding board member of the Entrepreneurs Foundation of Colorado, and Founder and CTO at Rally Software Development. Subscribe today to get free updates by email or RSS.

At Rally, we are always working on both maturing and growing our use of Agile. We started with a single development team and over the past 6 years have been through the process of splitting, growing, partnering, and acquiring.

We did this while continuing to inspect and adapt our development and our strategy execution processes.  We have teams in various stages of maturity using Scrum and Kanban to run all parts of our company.   As the CTO, I have my focus on our strategic planning and execution process.

In 2008, I started to focus on maturing our annual and quarterly planning.  To do that, I used Pascal Dennis’ book titled “Getting the Right Things Done” to structure those efforts.   (See related post about Learning from A3’s a story of 2009 Quarterly Planning at Rally.)

As part of that effort, we worked to change our quarterly and annual planning process to specifically follow a Plan-Do-Check-Adjust model. (Note that I like Pascal’s use of Adjust, not Act which is typically quoted in the Toyota models.)

Prior to 2009, we were simply using an inspect and adapt process to determine annual and quarterly priorities, aka quarterly rocks, based on feedback from last quarter’s results and the corporate roadmap.

This process worked well, but we had some issues including:

• A moving definition of done based on different standards of work (research, implementation, campaigns..)
• A delay in the feedback loop on strategic efforts made it hard to check and close well
• Too many efforts in a quarter lead to poor quality (We found 5 rocks to be too many for us during a quarter)

None of these are different than what most teams experience with going Agile.  So we (1) adjusted and moved to limit our WIP to three rocks, (2) focused on inspecting the definition of done in the meeting, (3) used company wide survey’s to keep from developing group think and (4) chose to do company celebration based on quarterly metrics and not the completion of quarterly rocks.

These all helped make the current process work, but the process was still frustratingly unpredictable, semi-random and always left something to be desired.  Many of the reasons for this are explained in Alan Shalloway’s and Don Reinertsen’s posts on PDCA and types of process The Difference Between “Inspect and Adapt” and Plan-Do-Check-Act (PDCA). Unlike Alan, I do not see or perceive a big issue with Scrum.  Based on my previous post around the roots of agile; Dean Leffingwell and I are in the same camp; Scrum is Lean.

As a result of moving to PDCA approach, we created a single “True-North” goal for the year and drove our quarterly rocks  towards that goal.

Now in Q4 of the year, we had some new changes to our process. By following the PDCA cycle for the year, we put a fine point on CHECK in this final quarter; Subsequently, we have a  Q4 quarterly rock focused solely on checking our Q3 and annual work to fine tune it based on real output.  This is an example of where PDCA cycle is more intentional than basic inspect and adapt at forcing the discipline of checking.

We focused a quarterly rock on checking  to make sure that we are done-done-done with our True North goal for the year.  We also have another Q4, cross-functional rock team focused on preparing for Q1 and 2010 annual planning.  This PDCA-driven rock is a major milestone for me personally.  It moves annual planning solely from my shoulders to a team effort; this pushes ownership of strategy down to the extended management team.  As a result, I am very happy with the move to PDCA for our 2009 strategy execution process. In Don Rienertsen’s terms,  our PDCA-driven process is more defined, while still with un-predictable output and governed with lots of feedback.   This was simply an increase in process maturity that was mandated by our continuing growth.

To do this, we create a team, called the Mountain team, to help the company transition our strategy execution process.   This team steered the transition and proposed our quarterly rocks based on the PDCA process.  And thanks to the ego-less and steady hand of our CEO, we have a very collaborative culture that quickly converge on these changes and quickly put them into action.

I hope this was helpful for you to learn about our experiences with continuous process improvement and our step-function transition processes.  Please note that we are not a perfect comparison to larger organizations trying to transition to large scale agility.  In addition to doing lots of growing, we have another difference that started when we began back in 2003.  We built Agile and Lean principles into our core values.  You can see the difference this might make in my comments to Israel Gat’s post More on Agile Social Contracts.

Specifically our core values are:

• Create your own reality
• Make and meet commitments
• Theory-driven decision making
• Treat people with respect
• Support our community and give back
• Maintain a healthy work/life balance

This is the social contract that we keep with employees. During transitions like this you need culture or a social contract to reinforce your moves toward Agile and Lean behaviors.

About the Author: Ryan Martens is a telemark skier,  founding board member of the Entrepreneurs Foundation of Colorado, and Founder and CTO at Rally Software Development. Subscribe today to get free updates by email or RSS.

I was teaching a CSM course a few months back when a question came up, as one often does, that needed an answer built around the concept of swarming.

An extremely creative example of the swarming concept

Swarming is something that is strangely alien to many folks in software development, so I’ll explain it here. Also, if I don’t explain it here then I won’t have enough to make a good blog post, and we can’t have that, can we?

The idea of swarming is to get the whole scrum team, or as much of the team as possible, to all jump onto a Product Backlog item (PBI) together and get it done in one fell swoop, as quickly and decisively as possible, by working together.

This has the benefit of being fun while at the same time implementing the idea of value-based delivery on the micro level. If a team swarms, it will tend to complete PBIs one after another rather than starting on several PBIs at a time and not completing any of them. So swarming on PBIs in priority order tends to deliver them in priority order while at the same time reducing the number of partially done PBIs (often called Work in Progress and considered to be a Bad Thing) that are hanging around waiting to be finished.

Swarming is something that good scrum teams do.

Side note: Those of you who are constantly asking what to do about the problem of testers not having anything to do until the last two days of the sprint, you should read this post twice. Those of you who are constantly carrying over unfinished PBIs to the next sprint, read this three times.

So, in response to the question, I told the story of swarming to the CSM kids as we sat around the campfire in the training room, eating smores, and they were enraptured. Entranced, really. Lightbulbs were flashing above heads. For some reason, for this class, the idea of swarming really hit home.

We had already discussed the Tuckman model of team dynamics earlier in the course. The Tuckman model, based on observations of lots of teams, simply lays out a four-stage pattern that teams seem to go through as they evolve.

The four stages of the Tuckman model are Forming, Storming, Norming, and Performing.

• Forming is fun because everything is new and nothing bad has happened yet. Of course, not a lot gets done during this stage, comparatively speaking.
• Storming is what happens as the team begins to try to get work done and the inevitable power struggles and turf wars begin. It’s not a particularly fun time, but it seems to be something that just happens when people try to work together.
• Norming occurs as the team resolves its internal strife and figures out how to work together.
• Performing is where the team can go when they learn to improve from their Norming plateau into a highly productive, smoothly-operating group of peer professionals.

The phrase “Forming, Storming, Norming, and Performing” is familiar to people who have been exposed to the Tuckman model for as little as 2.5 minutes.

So, in a burst of enthusiasm, one of the students in my class (remember the class?) suddenly shouted (well, to be honest, he only said it in a loud voice) “Forming, Storming, Norming, and Swarming!”

Which was great because Swarming is kinda one quick way to describe a Scrum team that is in the Performing phase.

Everybody laughed happily, and I thought, that’s a blog post right there. And look! I was right.

About the Author: Alan Atlas is a Soul Musician, Certified Scrum Trainer, and Agile Coach at Rally Software Development. Subscribe today to get free updates by email or RSS.