Agile Development Blog

This is #4 in a Series on the Culture of Innovation with guest blogger Lee Devin.

Do you notice a difference between problems and difficulties? A problem has a solution. When engineers solve it, the problem goes away. It’s a question raised for solution with fixes, tests, and checklist updates. In contrast, a difficulty has no solution. A difficulty wants you to sit with it, address it, not solve it. Artists know this world of the difficult very well.  No definitive fix, test, or checklist will suffice. Sitting with and playing with the difficult is simply part of the knowledge work of the artist.

For both the engineer and the artist, a difficulty is often what tangles up the solution to a problem. We see problems and difficulties all around us in the world of innovation. What’s needed to address a difficulty may not be clear at first, if ever. You may, in fact, never achieve that lovely industrial clarity. And yet, our ability to gaze unflinchingly into the face of difficulty will lead us to solve problems with greater innovation and deeper artistic mastery.

Difficulties require “AND” thinking

Difficulties are fuzzy. Improving how we address difficulties requires us to hold a large container with the word “AND” versus the word “OR.” This concept was first introduced to me in Peter Senge’s Fifth Discipline Fieldbook. I also delved into the topic in his course on Leading and Learning for Sustainability. To work with difficulties, we hold and play with a spectrum of possibilities, multiple solutions sets: this AND this AND this AND this. For the actors in a theater ensemble, AND means absorbing a variety of possibilities with the materials surrounding the play. Many innovative outcomes await based on the ensemble’s ability to hold the ambiguity of the art and embrace that sense of release.

For engineers, this might look like the following: you are solving multiple simultaneous equations for problems you see in the larger system. To be able to hold on to this container, you and the team have to feel safe “failing” with lots of little experiments. You must keep the “art of the possible” in mind. This is not an easy task for an individual, much less a group. Difficulties that accompany problems require the courage and patience to sit in a large container of ambiguity.

Consider the wicked problem example

In their book Wicked Problems, Righteous Solutions Peter DeGrace and Leslie Hulet Stahl help us delve further into problems and difficulties. Wicked problems arise out of several conditions. First, the problem domain is complex and fraught with difficulty. Then, the solution domain is similarly complex and difficult.  Finally the two overlap. That is a wicked problem. Wicked problems hold nests of difficulties. Let’s compare the wicked problem of an engineer and one of an actor.

The engineer’s work begins by reading a user story and exploring the problem sheet from the architecture council. In this assignment, the engineer must be prepared to address known and unknown difficulties on the path to a solution. The engineer recognizes that there is no one solution. How difficulties are addressed may be the key to just how innovative the resulting solution is.

This assignment is the equivalent of a script given to actors. The script is not a limit, but rather material on which to perform and interpret to create something new. There is no one solution. And so, the actors hold ambiguity as they move to the ultimate offering to their audience.  How the final play comes together may depend on the ensemble’s ability to play with and address the large realm of possibilities.

The art of the possible and innovation

Like actors, engineers and other knowledge workers need to do our homework, invite innovation and alternative solutions. We need to address difficulties not by point solutions, but by applying “AND” thinking, creating large containers of possibility. We must embrace the art of the possible. In the case of the actor, this comes in the form of rehearsal, ensemble and release that ultimately leads to the actual performance. In the case of the engineer, this work comes in the form of design spikes, set-based engineering, and tests. In both cases, experiments create space around difficulties. The art of the possible broadens the team’s or troupe’s innovative outcomes.

For such a culture of innovation, “AND” thinking is a vital function. At Rally, we apply “AND” thinking in how we address difficulties in a variety of ways. We may take a particular problem with its difficulties and spread it across a paired team of engineers. The teams work safely in an “art of the possible” mode for addressing difficulties in a way that leads to a better solution. We eventually move through the “AND” to a final solution, having addressed the difficulties in a variety of contexts. How you invite “AND” thinking and the art of the possible into your organization may include your Chief Engineer, a Product Owner, Director of System Engineering or a peer engineer all working in a new larger container.

Say “No,” to “No way!” and “Yes,” to “Imagine if…”

Within given circumstances, which are different for each team member, and a safe container for the conversation, the group can play out the implications of a solution, and discover its virtues and flaws. As with an ensemble of actors working through the possibilities of a scene, the only rule is: Never say, “No!” Swallowing that “No” can be hard! You must fight your first response to a suggestion or proposition, which is often “No, there is no way that it will work.”

Instead, the thing you must do is think, “Wait a minute. Let’s assume it will work. Let’s find out what happens when it does.” And, you’ll find out what happens when it does by behaving (thinking, talking, deciding) as if it is in place and working. If your first response is “Oh, wow, what a great idea!” the release might be, “How do I fit myself into this? What can I do personally to make it work?” The tool here is imagination.

We must find comfort in ambiguity and uncertainty. The question is: how can you create the container that allows your team to live with this difficulty and keep from jumping to try and solve it? If we create a culture friendly enough to notice accident and serendipity, we set ourselves up for the asymmetric payoffs associated with successful innovations. Our “AND” thinking and art of the possible ferret out the wicked problems and harvest collective team creativity.

Here’s something obvious I’ve learned the hard way: delivering “potentially shippable product increments” each and every Sprint isn’t easy.  Essentially you’re trying to take all the disparate activities that occur throughout the waterfall process, focus them on just the little product increment’s functionality and then jam them into a teensy little Sprint.  Hard to do and definitely pretty unlikely to get accomplished right out of the chute. The authors of “Practices for Scaling Lean & Agile Development”, Craig Larman and Bas Vodde, forgive you in advance for not getting this done straight away. In fact they suggest that this is more a goal the team will approach over time than a rule they put in place on day one.

The authors propose something very simple but very insightful:

• Sketch out all the things you need to do to get your product out the door.
• Define “done” as that subset of those the team is currently capable of performing every Sprint.
• Use your retrospectives to challenge the team to bring more and more of the “undone” work into each Sprint.

This has already changed the way I think about retrospectives. For other nuggets from Larman and Vodde’s book, read on.

Overall a Must-Read for Agile Development Leaders

I was blown away by “Scaling Lean & Agile Development”, as you can see from my bullish review on O’Reilly.  Some time has passed since then but I still feel that it’s one of the most important development books I’ve read.  That book alluded to the companion volume, “Practices for Scaling Lean & Agile Development”, and as you can imagine I awaited its publication eagerly.  It came out in February – I’ve worked my way through it now.   It’s most definitely a worthy successor.

The first book presents theoretical and philosophical underpinnings for agile and lean development. The second book presents a survey of practices relevant to all aspects of the process of developing software at scale, presented by two guys who bring a wealth of knowledge and experience to the table.

Continual Investment in Requirements

Larman and Vodde present practices relevant to a continual investment in requirements throughout the product development process. This is done both up-front, in seeding the product backlog, and iteratively, in refining requirements to support upcoming Sprints.

I love the emphasis they place on whole-team involvement in the initial Product Backlog development effort – even for projects with large teams.  Too often I’ve seen this be the provenance of Product Owners, working in near isolation, which does little to get the project off on a good footing.  The first the team sees of the requirements is the Product Backlog itself, having been involved in none of the  discussions and thought that went into it.

The notion of requirements areas, which was introduced in the first book, is leveraged here to help parallelize the initial requirements development work.  Concurrent sizing and value estimation workshops keep the requirements work rooted in reality.  They spend some time on the problem of bootstrapping a consistent sizing process in a large scale team. The use of cross-team estimation sessions result in the development of a canonical set of sized stories used as a basis for subsequent sizing at the team level.

In Favor of Forward-Looking Requirements Refinement

The discussion of forward-looking requirements refinement really resonated with me. I’ve found in my own practice that peeking ahead at the Product Backlog items upcoming in the next Sprint and then spending time together working through them to understand what they really mean – before we get into the Sprint planning session – goes a long way towards supporting a predictable iterative process.

It also puts the requirements elaboration just barely outside the Sprint that the work is planned for.  This separates coming to an understanding about what we want from the stress of figuring out how to fit it into a Sprint, which makes for more open and enjoyable requirements development.  The authors suggest the use of examples and Acceptance Test Driven Development (ATTD) to more precisely capture the intended behaviors of the stories in a way that realizes both conversation and confirmation of the “three Cs” . Smart.

The authors show how traditional approaches to project scoping and commitment (where separate product management and development organizations act essentially as competitors) are very much analogous to the contract negotiation that the Agile Manifesto cautions us against.  I’m surprised to admit this never occurred to me – I always read that part of the manifesto pretty literally. But it’s inarguably true that the wrangling between these two groups over project scope and timelines that happens in many organizations is a form of contract negotiation, and the waste it drives can be startling.

The arm wrestling of the product management “market ask” captured in a Market Requirements Document followed by the team’s “development response” carefully defined in a Product Requirements Document is a case in point.  Weeks and months can go by during this ritual.  What are the development teams doing during this time?  Not at lot, in my experience.  Does development really “win” if they manage to push out the end date or reduce scope?  Does product management really “win” if they manage to squeeze in extra features or pull in the release date?  I’ve seen how the empowerment of the Product Owner and the establishment of the Product Backlog as the single high-level requirements and release scoping artifact helped to prevent some of these painful dysfunctions but I’ve never thought of them in terms of an imbalance of contract negotiation over customer collaboration.

Creating “Desire Lines” in Design

Larman and Vodde strongly advocate wikis as the preferred basis for the technical documentation the team develops.  They suggest a page for the overall product and pages for each Sprint. That worked for my teams as well, although there was always a certain (healthy) tension regarding what was appropriate for Sprint documentation and what should be living documentation at the product level.

They present the compelling idea of “desire lines”, which, in landscaping, refer to paths that develop naturally, as people use a given space.  Rather than guessing how people would use the area, they are observed using it and then the landscaping is designed around their actual usage.  Similarly in design, rather than trying to guess what the needs are, let them emerge and then refactor to support the emerging design tensions.  A lovely analogy, I thought, and one that will stay with me.

They suggest both a priori design in collaborative design workshops and design after the fact in System Architecture Documentation (SAD) workshops.  I like the acceptance that both approaches are going to be useful. The former stresses the need for team contribution to the design as the Product Backlog items are being developed, while the latter recognizes that team needs for technical documentation will emerge over time and so setting time aside to fine tune design documentation is both warranted and healthy.

The authors also address the question of whether and when to rewrite code – like Joel Spolsky, I personally favor refactoring to improve legacy code rather then re-engineering.  The authors present a compelling and complementary argument for refactoring and incremental improvement based on the value of instilling a notion of continual (rather than punctuated) improvement in the development teams.  They stress that the real problem isn’t the legacy code you’ve got but rather the legacy code you continue to write.  Encouraging the team to have a mindset of each check-in leaving the code base better than it was before – fixing the broken windows and taking out the trash – is a better solution to the problem of poor code quality than is carving off large sections of time for exclusively improvement-oriented work.

Acceptance Test Driven Development

The book’s worth it just for this material alone.  Larman and Vodde present a wonderful analysis of ATDD and characterize its place in the context of Scrum, including tying it into iterative requirements refinement, the Definition of Done and the Sprint Review.  I’d seen teams go in this direction driven largely by the desire to better engage testers in their teams throughout the Sprint and avoid “scrummerfall.” The authors take this further to show how ATDD, in which acceptance tests are defined and automated in advance of the development of the code that will pass those tests, does for teams in an iteration what Test Driven Development (TDD) does for individual developers in ten minutes.

They stress the need for the test automation to be a distributed responsibility shared by the whole team rather than one assigned to a specialist team.  I couldn’t agree more.  I’ve seen many attempts to establish test automation through the creation of a group apart from development and I can’t say that I’d call any of them particularly successful – at least not in the broad and encompassing sense that Larman and Vodde are envisioning in this book.

Continuous Integration at Scale

I was particularly glad to see treatment of Continuous Integration (CI) at scale. I’ve seen groups start with vanilla CI as it is described in the Extreme Programming literature and do well with it initially but then fail as their groups grew larger.

Essentially, Larman and Vodde propose a nested set of continuous integration builds, each larger one subsuming sets of smaller ones within it and each build defined for a particular concern.  Examples of these concerns include the verification of specific components and subsystems but also particular kinds of testing – including things like performance and stability testing.  One key aspect of this approach is that, at each level, you’re trading off the immediacy of feedback to the developer against the likelihood of the developer’s submission affecting quality at that level and the expense of the tests.

The Elusive Definition of Done

As noted earlier, Larman and Vodde,  suggest defining “done” as that subset of the work needed to release the product the team is currently capable of performing each and every Sprint – then use your retrospectives to challenge the team to bring more and more of the “undone” work into each Sprint.  The authors point out that there are really only two ways to extend the Definition of Done:  increase the cross-functionality of the team or increase the degree of automation the team uses.

In the meantime, however, while the Definition of Done leaves some work undone, Larman and Vodde suggest meeting that problem honestly, by doing things like defining an undone work team and pushing undone work onto the Product Backlog.  By being explicit about where the team currently stands against the goal of delivering potentially shippable product increments, this undone work can be better managed and the team’s insights can be directed to the problem of expanding that definition of done to get them closer to the goal.

For the Bookshelf of any Agile Team Member

The book isn’t without its faults.  It’s certainly long enough and some sections can be tedious (there’s a fifteen or so page section where different scenarios for story splitting are laboriously addressed – I get that this common complaint about stories needs to be put to rest but this feels like killing me with kindness :)).  The book’s organization lends itself more to use as a reference than as something you’d read cover to cover.   There are many repetitive sections that allow each chapter to stand on its own but are a bit hard to get through when you read it straight through.  But these are really just quibbles.  In all, “Practices for Scaling Lean  & Agile Development” is, just as its companion volume before it was, a tremendous book that belongs on the bookshelf of any agile coach, manager or team member.

I’d like to thank Anne Greenhaw and Selaine Henriksen for their help in developing this post.  Thanks also to Ryan Martens for inviting me to post here.

About the Author: Ed Willis has been a ScrumMaster, Product Owner, Scrum coach and trainer.  He is currently a developer in the telecommunications industry.

Last week I had the pleasure of sitting down to breakfast with David Douglas, co-author of Citizen Engineer, and Bernard Amadei, founder of Engineers without Boarders.  It was great to get them both to meet and discuss the need for global, citizen engineers in this increasingly complex and interconnected world.  If you are an engineer and you have not seen or read David and Greg Papadopoulos’  handbook for socially responsible engineering, then you are missing a great picture of the future of engineering driven by purpose and the question “why?”.

To put it simply as possible, Citizen Engineers are the connection point between science and society – between pure knowledge and how it is used.  Citizen Engineers are techno-responsible, environmentally responsible, economically responsible, socially responsible participants in the engineering community.

- Citizen Engineer

I happened to catch Bernard on the way to speak to the National Academy of Engineering on “Engineering Sustainability in the Face of Natural Hazards.”  This brought us to the oil spill in the Gulf Coast.  If you buy the tenents of the Citizen Engineer, then an engineer would be the spokesperson for BP in a situation like this.  In that role, the Citizen Engineer would talk about the situation and help educate the public on the implications of technology of deep water drilling.  At breakfast, this conversation gained a bunch of energy and stimulated me to explore this idea more completely.

Based on my experience and ideas contained in the Citizen Engineer, I believe we need to create more Citizen Engineers. If this happens, we can jump quickly past the island of blame and towards faster learning and more constructive solutions. By moving to a more visible, open and collaborative discourse, we can work together to address these global and complex difficulties.  So, my new favorite phrase is, “What would the Citizen Engineer do?”

In a world of increasing complexity, accidents happen.  This accident is a tragedy with 11 dead and 17 injured in an explosion that created the worst oil spill in the history of the United States.  Let’s start the clock over on these events and explore what a Citizen Engineer would do.

Managing the Gulf Coast Oil Spill, the Citizen Engineer way

It is April 20, just after the blow out. The Citizen Engineer, holding the title Chief Engineer at the company, was notified immediately by email, text and phone.  Right away, she started a number of things in parallel. First, her office took control and governance of the situation and began acting as the general contractor for the accident. There were four fronts to work on:

NASA photo taken May 24 - from web site http://2010gulfoilspill.com/

• Root cause of explosion and rig stability
• Continuing leaks
• Spill clean-up at sea
• Spill clean-up on land

This Chief Engineer’s office placed lead engineers on all these fronts, but to illustrate the point of our story, we will focus only on efforts to stop the continuing leaks.

In the first 24 hours, her team classified the accident as a complex situation, beyond the solution scope of past accidents. It was classified as complex due to the depth of water, pressure, size and number of leaks and the state of the well including the stuck drilling rods.  It was clear that relief wells would be the correct long-term fix, but they were months away.  As a result, her team quickly realized that this complex situation required them to learn as fast as possible from as wide group of people and as many experiments as possible. Simply reaching to internal or known experts of past solutions in shallower, more straight-forward situations would be fine in a complicated situation, but the pre-conceived solutions could actually hurt in this situation. After meeting her response team on-site, she launched the following parallel efforts:

1. Opened communications to the world via Internet to communicate video and known conditions of the accident including live underwater video feeds, movies of experiments and well configurations.
2. Called for counter-measures ideas and technologies from the petroleum engineering community with special requests to Norway and Brazil, the two leading countries with deep water well expertise.
3. Set a daily cadence for coordinating status and learning inside her team.
4. Pulled well experts from their partners, Halliburton and Transocean to staff her disaster response team.
5. Procured the submarines and well capping equipment for these depths.
6. Developed a model of the underwater site to make communication about the situation more clear.
7. Authorized the drilling of relief wells for long-term containment.

By opening communication of the situation to the world and inviting engineering help via the Internet, her team encouraged a crowdsourcing and expert sourcing approach to the problem.  As a result, they quickly received estimates on the amount of oil leaking from scientists who were familiar with measuring flows simply based on the video feeds.  Having understood the large magnitude of this flow, the response team was able to garner more dollars to expedite experiments based on simple, back-of-the-napkin estimates of costs due to fines and clean-up that would accumulate each day the well leaked.

Simultaneously, the web site was collecting potential countermeasures from petroleum as well as civil and aeronautical engineers from around the world.  These countermeasures were filtered by the web team and small groups of response team engineers were doing quick research, experiments and models to boil up the most feasible and effective ones. A web-based social media voting and comment system was allowing outside engineers to validate their thinking.  As the most effective countermeasures emerged, the team started to describe experiments necessary to learn how to evaluate the valid sets of potential solutions. Using their growing resources, the response team launched multiple experiments using models and simulations to accelerate their Orient-Observe-Decide-Act loops. Based on what they understood, they took a set-based approach to running these experimental solutions under the sea.

At the end of the first 24-hour cycle, they were clear on the first three underwater efforts.  These efforts were quick, easy and non-destructive to other efforts. Within the next three days, their first experiments did not attempt to slow the leak, but they learn much more about the actual situation of the undersea drill rig, the actual leak size and mix of gas and oil. This data allowed them to update their models and again narrow their choices, as well as feed the root cause and leak containment teams some valuable facts. They were learning and now major equipment was starting to arrive at the site.  They chose to work on the quickest solutions that had the highest estimated effectiveness and least likelihood to ruin the well site for further efforts. All of these models, experiments, and solutions sets were published on the web site in real-time.  The web site formed the basis for governmental and public communication updates as well kept the worldwide crowd of paid and volunteer engineers in the loop.

This learning-first approach led to some quick wins that started to slow the leaks only 10 days after the accident and fully contained it 14 days later.  There was now an estimated 200,000 barrels in the water.  Her attentions turned to other teams. One had the long-term, relief well underway with an estimate of 2 more months to completely contain the band-aided well from other leaks. The results of the response teams efforts kept the total spill size to less than the 250,000 barrels spilled by the Valdez in 1989 and less than the 7 million barrels spilled during Katrina. The Chief Engineer’s teams had used all the best thinking and resources from around the world to narrow to a short-term fixes very quickly.

To conceptually “pay back” the world of volunteers and future deep sea oil teams, the problem sheets, experiment results and retrospective meeting notes are all freely available on the corporate web site.  This site and content are open and shared with the world in an open source manner.  These notes provide data for future Chief Engineering teams to reference during future accidents.  They also provide an engineering case study and market data for equipment suppliers to the petroleum industry to help make these kind of efforts safer in the future. They know that by working fast and leveraging all the world’s resources, they directly attacked the highest economical, ecological and social risk quickly.

Are you a Citizen Engineer?

Things are changing, as we are rightfully blurring the lines between economic, social and environmental responsibility.  Everyone is having to become more responsible to the triple bottom line.  In this new world, the Citizen Engineer needs to be responsible to technology, ecology, society and economics.  In many cases, the Citizen Engineer must acknowledge the difference between problems and difficulties. Problems have answers, but for the difficulties we can do nothing but try to address it in our increasingly large-scale, interconnected and complex world.

Who knows if this approach would lead to a smaller spill in the future, but it would certainly lead to faster learning in the next set of accidents.

How does your engineering team behave during your organization’s accidents?

Ryan Martens is a civil engineer, founding board member of the Entrepreneurs Foundation of Colorado, and CTO at Rally Software Development.