Agile Development Blog

Michael Hugos, principal at Center for Systems Innovation [c4si], writes, speaks and consults on strategies for IT and business agility and mentors development teams. He spent six years as CIO of a multibillion dollar distribution cooperative developing a suite of supply chain and business systems that transformed the company’s operations and revenue model. He won the CIO 100 Award and Premier 100 Award for his work. He’s author of several books and writes an online column for CIO magazine called “Doing Business in Real Time.” We recently met with Michael at the Agile 2010 conference, which resulted in “Agile is Ready for the Enterprise” and sparked the idea for this blog post.

Rally asks: What issues and trends are you seeing across technology departments, development teams and in discussions with CIOs?

Michael Hugos answers: The example set by companies such as Google, Facebook and Netflix shows how companies can use iterative development to continuously enhance products and grow market share. This is being noticed by business and technology leaders in other companies and they are asking if they can do the same thing to drive development in their own companies. People realize that IT is right down the middle of everything a company does, and that traditional software release cycles of once a year, or even once a quarter, are not able to keep up with the pace of change and innovation these days.

Just like the word “athlete,” the word “Agile” grabs your attention; it sounds great. But moving from desire to reality always tests peoples’ commitment. A lot of companies are struggling with the all-too-common reaction of, “That’s not the way we do things here…” Agile approaches are interesting and fascinating to companies, but then there is the tendency to immediately criticize new ideas – we’re all prone to it. As soon as someone suggests a new way of doing something, we all think of 10 reasons why that can’t be done or why it won’t work.

Rally: What is driving enterprise adoption of Agile?

MH: To begin with, agility is no longer just a good idea; it’s now backed by law – the law of probability. This law says if a company can’t keep up with rapid rates of change in the world then its probability of success is getting smaller and smaller every day. And since companies need IT infrastructure and applications to operate, just as our bodies need a nervous system and muscles to move, IT agility is critical for a company to achieve business agility.

In the last few years, software tools have enabled executives to measure and track progress on Agile projects and to see the performance of Agile teams in widely dispersed geographical locations. That makes Agile methods more feasible for large companies. A pervading feeling exists throughout business that just about everything else has been tried and IT groups are still not really keeping up with the backlog of user requests. Users are starting to go around IT and do their own things using SaaS, social media and mashups to put together systems. So why not give Agile a try?

Rally: How do Agile methodologies help large organizations foster, regain or accelerate the pace of innovation?

MH: Agile practices offer the best way to improve communication and collaboration between business and IT. Meaningful innovation always starts with communication and collaboration. Another thing that Agile practices enable is the ability to try out new ideas and explore opportunities quickly without investing a lot of money up front. With more traditional approaches, companies invest a lot of time and money planning up front before they start something new. This is expensive. And since most new ideas don’t pan out in the end, this traditional approach makes it difficult (if not impossible) for companies to try out enough new ideas in a year to find that small handful of ideas that do work out and deliver the profits they are looking for.

I like to say that in this high-change and unpredictable economic environment, companies need to: “Think big, start small and deliver quickly.” That’s the best way to keep up, adapt and respond to change, and find the opportunities they are looking for. Agility means letting go of slow, deliberate decision-making in favor of quick, repeatedly-tested decisions. That’s why Agile methods are so appropriate for energizing companies and helping them develop innovative products and services.

Rally: How do you make a case for Agile and address the fears of risk-averse CIOs, CTOs or CEOs?

MH: First, I remind executives of something that has become a fact in the last 10 years: business operations and technology are so tightly intertwined that there is no meaningful distinction left between the two; you can’t do business without technology. That might seem obvious to many but, executives who have been around for a while (like me) may still remember the days when IT was just a back office operation.

Once people acknowledge this reality then I point out that, over the last 10 years, Agile practices have been thoroughly field tested and have an impressive track record for delivering success. There are software tools now, like Rally and others, that address Agile project management and reporting, business and IT collaboration, software testing, and the continuous integration of new software with existing systems infrastructure. So going Agile is not just a leap of faith anymore.

Agile is actually a better way to manage risk versus using traditional waterfall approaches. With Agile practices, big projects are divided into lots of smaller projects that build on each other. This enables people to employ short feedback loops, learn quickly and change plans in light of new information. Two of the biggest causes for failure in business and failure in new development projects is that companies have no inexpensive way to investigate new opportunities, and they blindly follow predefined project plans without change – even as the world itself keeps changing.

The IT profession is at a turning point: one group of IT practitioners has learned that agility is the way to go, but more traditional practitioners still call it radical. Yet, the traditionalists continue to apply the same old ways of doing things that result in the same old horrendously expensive, multi-year projects that produce systems barely better than what was there before, if they even work at all. More and more business executives are coming to the conclusion that the effective support of business agility is the main reason for their company to have an internal IT group. Otherwise, there are options now to just outsource IT operations to cloud computing vendors and get new applications from SaaS providers and social media.

Rally: What does the future hold for Agile and Lean development practices?

MH: Probably the biggest change will be analogous to what happens when a company grows and transitions from an entrepreneurial startup to an established business. When this transition happens, there is a need to become more pragmatic and less idealistic. In the Agile world, this means that “Scrum-but” will actually be the best way for most companies to adopt Agile methods. Each company will customize versions of Agile that best fit their needs and it will be some combination of practices from Scrum, XP, Lean, Kanban, etc. Even waterfall practices have some benefits which should be incorporated where they make sense. Agile practices will not be set in concrete; they will continue to evolve over time as companies learn more and the world keeps changing.

Another big change for Agile is the realization that Agile development is not an end in itself. The value of IT agility is its ability to drive business agility. In the end, agility is more about business than about IT. Instead of co-locating business people with development teams, we will embed IT people in business operating units and co-locate development teams with business people.

I talk about this in my most recent book Business Agility: Sustainable Prosperity in a Relentlessly Competitive World. Agile companies will become real-time organizations that use IT to drive a process of continuous focusing on and responding to opportunities and threats. They will employ IT to drive three continuous feedback loops that make their real-time operations possible. The first feedback loop (I use the Yin-Yang symbol), provides awareness of a changing environment and identifies threats and opportunities. The second loop (I use a sunflower because of how it constantly adjusts itself to follow the sun across the sky), provides balance and continuously adjusts existing operations and processes to fit changing circumstances. And the third loop (I use the leaping panther), provides agility in the sense that it is how companies create new processes and products to seize new opportunities. The figure below illustrates this.

This week both Jean and I delivered talks on the Agile organization at Agile 2010 in Orlando. Whether you were able to attend one, both or neither, this post shares the handouts and materials that we used in the talks.

If you attended, please provide comments on what you liked, were puzzled by and might change in the future.

Jean’s work was a three-hour tutorial on learning models for managing the Agile organization.   She ran three exercises and provided a bibliography of books/resources that we have used here at Rally:

Jean in action at Agile 2010

• True North Example

• True North Exercise

• Strategy A3 Template

• ORID Exercise

• Personal and Corporate Visioning Exercise

• Sample Quarterly Agenda

• Bibliography

In addition to Jean’s talk, I presented an experience report on our use of Plan-Do-Check-Act (PDCA) at Rally.  This report tells a story of our evolution of strategy execution from Gazelles/Scrum to Lean/Agile.

We hope these resources provide you with ideas for scaling your own Agile efforts beyond their current levels.  Again, please comment on the blog with what you got from the materials or the talks.  We want to hear from you on this topic.

Ryan Martens is a tomato grower, founding board member of the Entrepreneurs Foundation of Colorado, and CTO at Rally Software Development.

There are other conferences that cover Agile software development, but the Agile 20xx show reigns supreme. At nearly 2000 attendees from around the world, this year’s show is happening at Walt Disney World in Orlando.  (It was moved there after the flood in Nashville.) For the first time, three of the major analyst firms (and as a result 5 of  the key analysts who cover Agile and ALM) are attending the conference – Forrester, Gartner and IDC.

Rally coaches, sales and marketing folk at booth setup

As a result of the show’s success, it has become the most significant market rhythm in our industry.  So this week, we announced a few things:

• Rally Software Extends Agile ALM Platform to Meet the Unique Needs of Global Organizations – New customization options, full traceability with build and source code, and sharing with corporate dashboards create a single source of record for progress and quality
• Rally Software Extends Visualization of AgileZen Kanban Board – New non-profit program offers a 50% discount on leading visual project collaboration tool
• Innovation Games Company Announces Integration with Rally Software – The leader in serious games for business brings customers improved collaboration and prioritization capabilities
• Tasktop Introduces Agile Planner for Eclipse – New product enables Agile planning from the IDE and integrates multi-vendor Agile ALM solutions

I am speaking tomorrow on PDCA: Moving Beyond Simple Inspect and Adapt. (Thurs 9:00 a.m. in A-1). Other Rally speakers remaining this week are:

Get your Rally cap

• Former Rally developer turned Rally technical account manager turned Rally coach Chris Browne speaks Wednesday on The Art of the Hackathon (Weds 15:30 – 17:00 in Asia 3).
• Rally coaches Alan Atlas and John Martin speak Thursday on “Your Team, Your Freedom, Your Responsibility” (Thurs 15:30-17:00 in Asia 3).

Follow the news from the show on Twitter at #Agile2010. Come see us at the booth and get a Rally and Deliver ballcap. Or let us know if you’re not at the show and want us to send you one (send name and address to kcaraway@rallydev.com).

Ryan Martens is an organic farmer, founding board member of the Entrepreneurs Foundation of Colorado, and CTO at Rally Software Development.

Ah, Marketing.  Enter a world of countless project requests, numerous stakeholders, limited resources and rapidly changing market conditions.

Sound familiar?  In fact, marketers face a lot of the same challenges as development teams, and Agile can be a powerful way to alleviate those common issues and intelligently plan our work.

In steps 1-5, I’ll explain how Rally’s marketing team conducts our version of Release Planning.  In steps 6-10, I’ll explain how we run our iterations to meet those commitments. Our planning processes continue to evolve, though this method has worked for awhile now.

10 Steps to Successful Marketing using Agile and Lean Practices

1. We recognize that Marketing has challenges that are different from Development.

• There is no unique product owner.  For example, if we chose Sales, then we would always rank lead generation over branding, customer programs or analyst relations, and that could ultimately hurt our company. Therefore we have to use some best-guessing to prioritize our backlog and determine what is most important.
• We face hard event deadlines set far into the future.  Sometimes we have no choice but to commit to an event or sign a contract months ahead of time.
• Each team member has unique expertise, i.e. writing, event planning, PHP development and so forth.  So one shared backlog is inefficient.

Now that we’ve reviewed the challenges, we give ourselves permission to do what we need to do, have patience and adjust anything that isn’t working for us.

2. Conduct an ORID to learn from the past

Before planning for the next quarter, we hold a retrospective in the form of an ORID, “a means to analyze facts and feelings, to ask about implications and to make decisions intelligently”, a process created by the Institute of Cultural Affairs.  We gather as a team to share:

• Observations (O) – What do we know?
• Reflections (R) – How do we feel about this?
• Interpretations (I) – What does it mean for the organization?
• Decisions (D) – What are we going to do?

This strategic overview helps set context for us to prioritize our focus for next quarter.

3. Align ORID decisions with company strategy

At Rally, we conduct quarterly and annual planning using the Plan Do Check Adjust methodology as explained in Getting the Right Things Done.   As we look at the overall company direction and goals, we keep these in mind as we hold planning at our own level.   Ideally, our major commitments support and align with company strategy. This also helps inform our “stop doing” list.

4. Poll our stakeholders

As part of determining quarterly commitments, we poll our major stakeholders for their top requests.  We use a Google survey to rank these requests by importance, size each request and bring these epics into our release planning meeting, to be included as part of our ranked backlog.

5. Conduct Release Planning to prioritize and agree on quarterly commitments

Now that we have all of our inputs, we hold our quarterly Release Planning session.  We write each epic on a sticky note and look at all of the possible work we could do this quarter.  Then, we evaluate epics based on importance taking company goals, stakeholder wishes, market realities like conferences and our own passions into consideration. We decide what we can realistically commit to, and agree as a team.  We keep in mind that making and meeting commitments is a huge deal, and we try really hard not to over-commit.

Part 2 – Meet our Marketing Commitments

6. Create a task board

Since our marketing team is mostly co-located, we pin up several large sheets of paper to use as a task board.  This is where we review our commitments on a daily basis as a sanity check that our stories are prioritized correctly and that we are tackling the right work as the quarter progresses.

As a team, we write our quarterly commitments on the task board with the definition of done assigned to each one.  We also include our “foundational” work – recurring work like website updates, online ad campaigns, field events, press releases and other important work that we need time to do.

Note: this is not a Kanban board because we do not have a shared backlog nor do we have a team-wide WIP limit.  As we break into smaller project teams that do share a backlog, we often use AgileZen to manage this work.

7. Hold iteration planning every 2 weeks

Every 2 weeks, we hold an Iteration Planning meeting.  Each team member has her own sticky note color, creates stories on those notes and manages her own prioritized backlog using T-shirt sizing to roughly estimate each story.  In this hour-long meeting, we begin by expressing appreciation for team members who gave exceptional assistance.  Then we hold a brief retrospective on what worked and what should change for the next iteration.  Finally, we each read out our prioritized stories for the iteration, putting them on the task board’s backlog.  This gives everyone visibility to what’s happening, identifies if someone is over-committed and lets the team swarm any epics with looming deadlines.

8. Conduct a daily stand-up

At the same time each day, we hold a stand-up meeting (with a consistent conference call #) that is at most 10-15 minutes long.  We form a semi-circle in front of our task board and share no more than 2 cross-cutting significant actions or take-aways from the day before, no more than 2 that we are planning to accomplish that day, and whether our work is blocked by any issues beyond our control.   As we start working on stories throughout the iteration, we move them from the backlog into their section of the task board to show what we are working on.  When the story is complete, then we move it to a place on the task board labeled “Done”.  Once the commitment’s Definition of Done is met, we check off that commitment and feel good about completing it.

9. Be patient as things change

It would be lovely if nothing changed during the iteration, but that just doesn’t happen.  The goal is ultimately to respond to change rather than cling to an outdated plan.  As new opportunities arise, new time-sensitive information appears and new requests are made, so our iteration work changes and that’s ok.  We try to just document what we’re working on and create new stories so that we can make intelligent prioritization decisions during the course of the iteration.

10. Retrospect, inspect and adapt

As we keep running our iterations and fulfilling our commitments, we are always looking for ways to improve them.   Ultimately, we’re using Agile to improve the quality of our work life by using objective, smart ways of planning how we spend our time. And we’re learning a lot from the journey.

I am passionate about design; if it were not for the boom-bust cycles in architecture, I would have followed that education/career path.  As a result of that passion, I got really excited when I saw HiveLive four years ago.  So excited in fact, that Rally jumped in as key first customer and based Agile Commons on HiveLive’s platform.  I even personally invested in the effort led by three Kembel brothers: John, Jeremy and Geoff.  Last year, HiveLive’s  journey took another turn as they sold to RightNow. After meeting RightNow’s David Vap and speaking with a good part of their technical team,  I would say John, the VP of Social Solutions, is right that they made a great move.

John’s design-thinking approach was front and center to HiveLive.  It came from his background at Standford’s design school and a stint at IDEO.  As I got to know John, he mentioned all the great things going on with his other brother and twin George.  George was busy creating a new version of the design school, called the Stanford d.school.   The new d.school has broadened beyond just a partnership of art and mechanical engineering to become a interdisciplinary school that brings design thinking to all majors.   As I learned more about this, I started pulling on John’s shirt to get me out there so I could go see the place and meet George.  Well last week, I participated in the first ever d.social summit for two days with 15 folks focused on the intersection of design thinking and social thinking.  Twins John and George Kembel actually facilitate in stereo.  To watch and be a part of their combined effort was like drinking from a fire hose.

The event and the people were great fun to work with and pushed my limits on the overlap of design thinking and social thinking.  Working there really made me feel strong and I found myself in a flow most of the time.  It caused me to notice that I really love the expansion of design to design thinking. But for you and your agile teams, the innovations in team room furniture was really important. Creating a culture of innovation relies on creating the right environment.  If you have read Takeuechi and Nonaka’s book on Knowledge Management, you will recognize the concept of “Ba.” Ba is the shared space that creates context for the knowledge-creating company.  (See figure 4.3 on page 102 of their book for a cool illustration of the whole concept)

The d.school is full of flexible, collaborative space of all kinds, shapes and sizes.  They are constantly trying new things there.  Built for running multiple, parallel design projects in 15 week cycles, it is empathetic to extreme users.  The space is in its sixth iteration of the space.  Scott Doorley and Dave Baggeroer worked with George over the last five years to really make this place something special.   As a result of working at the extreme of rapid collaboration, they have come up with some fantastic furniture designs that you should consider copying for your team and meeting rooms.  Unfortunately, you can’t buy this stuff – you have to build it locally.

Here is a set of stackable and highly portable white boards.

Notice the Z-shaped foot that allows them to stack and move around corners.  These ideas came from retail stores.  Also notice the red peg in the whiteboard.  This is designed to hang portable whiteboards that you can take back to your own space.   It could also hold a pad of flip-chart paper.

This line is where they store the student efforts.  Notice how the hanging whiteboards are stored.  It is easy to imagine collaborating in another person’s office and then bringing the whiteboard back to your office without using tons of flip chart pads.

Below is a portable wall system built with spring-loaded feet to allow you to make semi-transparent or opaque walls by lightly snapping them into place.  You can see them used above to make the line where student’s store their work.

These cool pommel horses, pictured below, make great furniture for a team collaboration space.  You can sit, stand or work at the structures and they force you to not think in hierarchies:)

Finally, don’t miss the d.school’s blog and the coverage of their sugar cubes.

I hope some of these pieces of furniture compel you to try some new furniture in your space.  If you are not quite sold, you might read Tim Brown’s new book “Change by Design.” It is a great living example of the approaches that IDEO and the d.school use to create empathy, insight and desirable design in physical, virtual and social systems.

Do you have any experience applying design thinking in your agile teams?  Jared Spool’s talk at Agile 2009 was a great example of applying design thinking to software.

Ryan Martens is a tomato grower, founding board member of the Entrepreneurs Foundation of Colorado, and CTO at Rally Software Development.

Matt Phillips is a Senior Project Manager who has spent the last few years helping shape the Agile development process at Lulu.com. He currently heads up the Lulu Project Management Office and has spent several months setting up Agile practices in Lulu’s India office, based in Bangalore. In advance of his executive panel discussion at Rally’s Agile Success Tour in Raleigh, NC, we sat down with Matt to ask him 5 questions about Agile.

1. How have you implemented Agile across your organization?

We’ve rolled Agile out among all of our distributed teams, which are located in Raleigh, NC, the Ukraine and India. The time zones have historically been a challenge, so we had our remote teams spend several weeks in the Raleigh office working through daily Scrums. Now, they’re essentially as included in the process as possible. We use video conferencing for daily Scrums and to schedule iteration planning. All the teams collaborate to define stories, determine velocity, and plan iterations. We use Rally to make projections, track our velocity, and get visibility into the health of our projects. The metrics have become indispensible for judging how we’re doing, making accurate projections, and delivering upon our commitments.

2. What was your #1 reason for adopting Agile development?

Lulu adopted Agile at a point where the company was very much in start-up mode. The ideas were coming at a frenetic pace and the engineering team size was poised to expand. Agile methodologies were a good fit for Lulu’s culture and environment. The concepts of short iterations and regular release cycles paired with Scrum provided a quick time-to-market period for new ideas. At the same time, by adopting Agile methodologies, Lulu gained increased insight into the development team’s progress and performance as the team grew and feature sets became more complex.

3. What has been the biggest benefit of adopting Agile?

The metrics and amazing visibility we have into development projects. This is especially important for a team that’s 9,000 miles away. We have visibility into how they’re progressing on features, what’s coming next in the roadmap, and really flushing out what the product backlog looks like and where we’re headed.  Prior to implementing Agile, it was very hard to sync-up  (because of the 9 ½ hour time difference), maintain a feedback loop and foster collaboration with teams so far away.

4. What one piece of advice would you give to new Agile teams?

My advice would be to ease into it – kind of like steering a cruise ship, not turning on a dime. Start with familiar concepts and gradually introduce Agile practices over time. We started with familiar ideas like release dates, associated task lists, estimations, and tracking criteria. Then, we used a phased approach to introduce the concepts of iterations, story points and relative sizing, velocity, and ranking. We continue to work toward more granular inputs to smoothly coordinate roles, tools and dependencies within Rally as we go along to continuously perform at higher levels and get better outputs.

5. How can you tell that Agile is successful at Lulu.com?

One of top ways I can tell that Agile is successful in our organization is that people, even outside of engineering, are speaking in story points. That tells me that Agile has really taken hold. Using story points and velocity for our release planning makes it easy to arrive at a date that everyone is comfortable with. On top of that, our track record since adopting Agile shows that we’ve been delivering on our commitments every time.

This is #4 in a Series on the Culture of Innovation with guest blogger Lee Devin.

Do you notice a difference between problems and difficulties? A problem has a solution. When engineers solve it, the problem goes away. It’s a question raised for solution with fixes, tests, and checklist updates. In contrast, a difficulty has no solution. A difficulty wants you to sit with it, address it, not solve it. Artists know this world of the difficult very well.  No definitive fix, test, or checklist will suffice. Sitting with and playing with the difficult is simply part of the knowledge work of the artist.

For both the engineer and the artist, a difficulty is often what tangles up the solution to a problem. We see problems and difficulties all around us in the world of innovation. What’s needed to address a difficulty may not be clear at first, if ever. You may, in fact, never achieve that lovely industrial clarity. And yet, our ability to gaze unflinchingly into the face of difficulty will lead us to solve problems with greater innovation and deeper artistic mastery.

Difficulties require “AND” thinking

Difficulties are fuzzy. Improving how we address difficulties requires us to hold a large container with the word “AND” versus the word “OR.” This concept was first introduced to me in Peter Senge’s Fifth Discipline Fieldbook. I also delved into the topic in his course on Leading and Learning for Sustainability. To work with difficulties, we hold and play with a spectrum of possibilities, multiple solutions sets: this AND this AND this AND this. For the actors in a theater ensemble, AND means absorbing a variety of possibilities with the materials surrounding the play. Many innovative outcomes await based on the ensemble’s ability to hold the ambiguity of the art and embrace that sense of release.

For engineers, this might look like the following: you are solving multiple simultaneous equations for problems you see in the larger system. To be able to hold on to this container, you and the team have to feel safe “failing” with lots of little experiments. You must keep the “art of the possible” in mind. This is not an easy task for an individual, much less a group. Difficulties that accompany problems require the courage and patience to sit in a large container of ambiguity.

Consider the wicked problem example

In their book Wicked Problems, Righteous Solutions Peter DeGrace and Leslie Hulet Stahl help us delve further into problems and difficulties. Wicked problems arise out of several conditions. First, the problem domain is complex and fraught with difficulty. Then, the solution domain is similarly complex and difficult.  Finally the two overlap. That is a wicked problem. Wicked problems hold nests of difficulties. Let’s compare the wicked problem of an engineer and one of an actor.

The engineer’s work begins by reading a user story and exploring the problem sheet from the architecture council. In this assignment, the engineer must be prepared to address known and unknown difficulties on the path to a solution. The engineer recognizes that there is no one solution. How difficulties are addressed may be the key to just how innovative the resulting solution is.

This assignment is the equivalent of a script given to actors. The script is not a limit, but rather material on which to perform and interpret to create something new. There is no one solution. And so, the actors hold ambiguity as they move to the ultimate offering to their audience.  How the final play comes together may depend on the ensemble’s ability to play with and address the large realm of possibilities.

The art of the possible and innovation

Like actors, engineers and other knowledge workers need to do our homework, invite innovation and alternative solutions. We need to address difficulties not by point solutions, but by applying “AND” thinking, creating large containers of possibility. We must embrace the art of the possible. In the case of the actor, this comes in the form of rehearsal, ensemble and release that ultimately leads to the actual performance. In the case of the engineer, this work comes in the form of design spikes, set-based engineering, and tests. In both cases, experiments create space around difficulties. The art of the possible broadens the team’s or troupe’s innovative outcomes.

For such a culture of innovation, “AND” thinking is a vital function. At Rally, we apply “AND” thinking in how we address difficulties in a variety of ways. We may take a particular problem with its difficulties and spread it across a paired team of engineers. The teams work safely in an “art of the possible” mode for addressing difficulties in a way that leads to a better solution. We eventually move through the “AND” to a final solution, having addressed the difficulties in a variety of contexts. How you invite “AND” thinking and the art of the possible into your organization may include your Chief Engineer, a Product Owner, Director of System Engineering or a peer engineer all working in a new larger container.

Say “No,” to “No way!” and “Yes,” to “Imagine if…”

Within given circumstances, which are different for each team member, and a safe container for the conversation, the group can play out the implications of a solution, and discover its virtues and flaws. As with an ensemble of actors working through the possibilities of a scene, the only rule is: Never say, “No!” Swallowing that “No” can be hard! You must fight your first response to a suggestion or proposition, which is often “No, there is no way that it will work.”

Instead, the thing you must do is think, “Wait a minute. Let’s assume it will work. Let’s find out what happens when it does.” And, you’ll find out what happens when it does by behaving (thinking, talking, deciding) as if it is in place and working. If your first response is “Oh, wow, what a great idea!” the release might be, “How do I fit myself into this? What can I do personally to make it work?” The tool here is imagination.

We must find comfort in ambiguity and uncertainty. The question is: how can you create the container that allows your team to live with this difficulty and keep from jumping to try and solve it? If we create a culture friendly enough to notice accident and serendipity, we set ourselves up for the asymmetric payoffs associated with successful innovations. Our “AND” thinking and art of the possible ferret out the wicked problems and harvest collective team creativity.

Here’s something obvious I’ve learned the hard way: delivering “potentially shippable product increments” each and every Sprint isn’t easy.  Essentially you’re trying to take all the disparate activities that occur throughout the waterfall process, focus them on just the little product increment’s functionality and then jam them into a teensy little Sprint.  Hard to do and definitely pretty unlikely to get accomplished right out of the chute. The authors of “Practices for Scaling Lean & Agile Development”, Craig Larman and Bas Vodde, forgive you in advance for not getting this done straight away. In fact they suggest that this is more a goal the team will approach over time than a rule they put in place on day one.

The authors propose something very simple but very insightful:

• Sketch out all the things you need to do to get your product out the door.
• Define “done” as that subset of those the team is currently capable of performing every Sprint.
• Use your retrospectives to challenge the team to bring more and more of the “undone” work into each Sprint.

This has already changed the way I think about retrospectives. For other nuggets from Larman and Vodde’s book, read on.

Overall a Must-Read for Agile Development Leaders

I was blown away by “Scaling Lean & Agile Development”, as you can see from my bullish review on O’Reilly.  Some time has passed since then but I still feel that it’s one of the most important development books I’ve read.  That book alluded to the companion volume, “Practices for Scaling Lean & Agile Development”, and as you can imagine I awaited its publication eagerly.  It came out in February – I’ve worked my way through it now.   It’s most definitely a worthy successor.

The first book presents theoretical and philosophical underpinnings for agile and lean development. The second book presents a survey of practices relevant to all aspects of the process of developing software at scale, presented by two guys who bring a wealth of knowledge and experience to the table.

Continual Investment in Requirements

Larman and Vodde present practices relevant to a continual investment in requirements throughout the product development process. This is done both up-front, in seeding the product backlog, and iteratively, in refining requirements to support upcoming Sprints.

I love the emphasis they place on whole-team involvement in the initial Product Backlog development effort – even for projects with large teams.  Too often I’ve seen this be the provenance of Product Owners, working in near isolation, which does little to get the project off on a good footing.  The first the team sees of the requirements is the Product Backlog itself, having been involved in none of the  discussions and thought that went into it.

The notion of requirements areas, which was introduced in the first book, is leveraged here to help parallelize the initial requirements development work.  Concurrent sizing and value estimation workshops keep the requirements work rooted in reality.  They spend some time on the problem of bootstrapping a consistent sizing process in a large scale team. The use of cross-team estimation sessions result in the development of a canonical set of sized stories used as a basis for subsequent sizing at the team level.

In Favor of Forward-Looking Requirements Refinement

The discussion of forward-looking requirements refinement really resonated with me. I’ve found in my own practice that peeking ahead at the Product Backlog items upcoming in the next Sprint and then spending time together working through them to understand what they really mean – before we get into the Sprint planning session – goes a long way towards supporting a predictable iterative process.

It also puts the requirements elaboration just barely outside the Sprint that the work is planned for.  This separates coming to an understanding about what we want from the stress of figuring out how to fit it into a Sprint, which makes for more open and enjoyable requirements development.  The authors suggest the use of examples and Acceptance Test Driven Development (ATTD) to more precisely capture the intended behaviors of the stories in a way that realizes both conversation and confirmation of the “three Cs” . Smart.

The authors show how traditional approaches to project scoping and commitment (where separate product management and development organizations act essentially as competitors) are very much analogous to the contract negotiation that the Agile Manifesto cautions us against.  I’m surprised to admit this never occurred to me – I always read that part of the manifesto pretty literally. But it’s inarguably true that the wrangling between these two groups over project scope and timelines that happens in many organizations is a form of contract negotiation, and the waste it drives can be startling.

The arm wrestling of the product management “market ask” captured in a Market Requirements Document followed by the team’s “development response” carefully defined in a Product Requirements Document is a case in point.  Weeks and months can go by during this ritual.  What are the development teams doing during this time?  Not at lot, in my experience.  Does development really “win” if they manage to push out the end date or reduce scope?  Does product management really “win” if they manage to squeeze in extra features or pull in the release date?  I’ve seen how the empowerment of the Product Owner and the establishment of the Product Backlog as the single high-level requirements and release scoping artifact helped to prevent some of these painful dysfunctions but I’ve never thought of them in terms of an imbalance of contract negotiation over customer collaboration.

Creating “Desire Lines” in Design

Larman and Vodde strongly advocate wikis as the preferred basis for the technical documentation the team develops.  They suggest a page for the overall product and pages for each Sprint. That worked for my teams as well, although there was always a certain (healthy) tension regarding what was appropriate for Sprint documentation and what should be living documentation at the product level.

They present the compelling idea of “desire lines”, which, in landscaping, refer to paths that develop naturally, as people use a given space.  Rather than guessing how people would use the area, they are observed using it and then the landscaping is designed around their actual usage.  Similarly in design, rather than trying to guess what the needs are, let them emerge and then refactor to support the emerging design tensions.  A lovely analogy, I thought, and one that will stay with me.

They suggest both a priori design in collaborative design workshops and design after the fact in System Architecture Documentation (SAD) workshops.  I like the acceptance that both approaches are going to be useful. The former stresses the need for team contribution to the design as the Product Backlog items are being developed, while the latter recognizes that team needs for technical documentation will emerge over time and so setting time aside to fine tune design documentation is both warranted and healthy.

The authors also address the question of whether and when to rewrite code – like Joel Spolsky, I personally favor refactoring to improve legacy code rather then re-engineering.  The authors present a compelling and complementary argument for refactoring and incremental improvement based on the value of instilling a notion of continual (rather than punctuated) improvement in the development teams.  They stress that the real problem isn’t the legacy code you’ve got but rather the legacy code you continue to write.  Encouraging the team to have a mindset of each check-in leaving the code base better than it was before – fixing the broken windows and taking out the trash – is a better solution to the problem of poor code quality than is carving off large sections of time for exclusively improvement-oriented work.

Acceptance Test Driven Development

The book’s worth it just for this material alone.  Larman and Vodde present a wonderful analysis of ATDD and characterize its place in the context of Scrum, including tying it into iterative requirements refinement, the Definition of Done and the Sprint Review.  I’d seen teams go in this direction driven largely by the desire to better engage testers in their teams throughout the Sprint and avoid “scrummerfall.” The authors take this further to show how ATDD, in which acceptance tests are defined and automated in advance of the development of the code that will pass those tests, does for teams in an iteration what Test Driven Development (TDD) does for individual developers in ten minutes.

They stress the need for the test automation to be a distributed responsibility shared by the whole team rather than one assigned to a specialist team.  I couldn’t agree more.  I’ve seen many attempts to establish test automation through the creation of a group apart from development and I can’t say that I’d call any of them particularly successful – at least not in the broad and encompassing sense that Larman and Vodde are envisioning in this book.

Continuous Integration at Scale

I was particularly glad to see treatment of Continuous Integration (CI) at scale. I’ve seen groups start with vanilla CI as it is described in the Extreme Programming literature and do well with it initially but then fail as their groups grew larger.

Essentially, Larman and Vodde propose a nested set of continuous integration builds, each larger one subsuming sets of smaller ones within it and each build defined for a particular concern.  Examples of these concerns include the verification of specific components and subsystems but also particular kinds of testing – including things like performance and stability testing.  One key aspect of this approach is that, at each level, you’re trading off the immediacy of feedback to the developer against the likelihood of the developer’s submission affecting quality at that level and the expense of the tests.

The Elusive Definition of Done

As noted earlier, Larman and Vodde,  suggest defining “done” as that subset of the work needed to release the product the team is currently capable of performing each and every Sprint – then use your retrospectives to challenge the team to bring more and more of the “undone” work into each Sprint.  The authors point out that there are really only two ways to extend the Definition of Done:  increase the cross-functionality of the team or increase the degree of automation the team uses.

In the meantime, however, while the Definition of Done leaves some work undone, Larman and Vodde suggest meeting that problem honestly, by doing things like defining an undone work team and pushing undone work onto the Product Backlog.  By being explicit about where the team currently stands against the goal of delivering potentially shippable product increments, this undone work can be better managed and the team’s insights can be directed to the problem of expanding that definition of done to get them closer to the goal.

For the Bookshelf of any Agile Team Member

The book isn’t without its faults.  It’s certainly long enough and some sections can be tedious (there’s a fifteen or so page section where different scenarios for story splitting are laboriously addressed – I get that this common complaint about stories needs to be put to rest but this feels like killing me with kindness :)).  The book’s organization lends itself more to use as a reference than as something you’d read cover to cover.   There are many repetitive sections that allow each chapter to stand on its own but are a bit hard to get through when you read it straight through.  But these are really just quibbles.  In all, “Practices for Scaling Lean  & Agile Development” is, just as its companion volume before it was, a tremendous book that belongs on the bookshelf of any agile coach, manager or team member.

I’d like to thank Anne Greenhaw and Selaine Henriksen for their help in developing this post.  Thanks also to Ryan Martens for inviting me to post here.

About the Author: Ed Willis has been a ScrumMaster, Product Owner, Scrum coach and trainer.  He is currently a developer in the telecommunications industry.

Last week I had the pleasure of sitting down to breakfast with David Douglas, co-author of Citizen Engineer, and Bernard Amadei, founder of Engineers without Boarders.  It was great to get them both to meet and discuss the need for global, citizen engineers in this increasingly complex and interconnected world.  If you are an engineer and you have not seen or read David and Greg Papadopoulos’  handbook for socially responsible engineering, then you are missing a great picture of the future of engineering driven by purpose and the question “why?”.

To put it simply as possible, Citizen Engineers are the connection point between science and society – between pure knowledge and how it is used.  Citizen Engineers are techno-responsible, environmentally responsible, economically responsible, socially responsible participants in the engineering community.

- Citizen Engineer

I happened to catch Bernard on the way to speak to the National Academy of Engineering on “Engineering Sustainability in the Face of Natural Hazards.”  This brought us to the oil spill in the Gulf Coast.  If you buy the tenents of the Citizen Engineer, then an engineer would be the spokesperson for BP in a situation like this.  In that role, the Citizen Engineer would talk about the situation and help educate the public on the implications of technology of deep water drilling.  At breakfast, this conversation gained a bunch of energy and stimulated me to explore this idea more completely.

Based on my experience and ideas contained in the Citizen Engineer, I believe we need to create more Citizen Engineers. If this happens, we can jump quickly past the island of blame and towards faster learning and more constructive solutions. By moving to a more visible, open and collaborative discourse, we can work together to address these global and complex difficulties.  So, my new favorite phrase is, “What would the Citizen Engineer do?”

In a world of increasing complexity, accidents happen.  This accident is a tragedy with 11 dead and 17 injured in an explosion that created the worst oil spill in the history of the United States.  Let’s start the clock over on these events and explore what a Citizen Engineer would do.

Managing the Gulf Coast Oil Spill, the Citizen Engineer way

It is April 20, just after the blow out. The Citizen Engineer, holding the title Chief Engineer at the company, was notified immediately by email, text and phone.  Right away, she started a number of things in parallel. First, her office took control and governance of the situation and began acting as the general contractor for the accident. There were four fronts to work on:

NASA photo taken May 24 - from web site http://2010gulfoilspill.com/

• Root cause of explosion and rig stability
• Continuing leaks
• Spill clean-up at sea
• Spill clean-up on land

This Chief Engineer’s office placed lead engineers on all these fronts, but to illustrate the point of our story, we will focus only on efforts to stop the continuing leaks.

In the first 24 hours, her team classified the accident as a complex situation, beyond the solution scope of past accidents. It was classified as complex due to the depth of water, pressure, size and number of leaks and the state of the well including the stuck drilling rods.  It was clear that relief wells would be the correct long-term fix, but they were months away.  As a result, her team quickly realized that this complex situation required them to learn as fast as possible from as wide group of people and as many experiments as possible. Simply reaching to internal or known experts of past solutions in shallower, more straight-forward situations would be fine in a complicated situation, but the pre-conceived solutions could actually hurt in this situation. After meeting her response team on-site, she launched the following parallel efforts:

1. Opened communications to the world via Internet to communicate video and known conditions of the accident including live underwater video feeds, movies of experiments and well configurations.
2. Called for counter-measures ideas and technologies from the petroleum engineering community with special requests to Norway and Brazil, the two leading countries with deep water well expertise.
3. Set a daily cadence for coordinating status and learning inside her team.
4. Pulled well experts from their partners, Halliburton and Transocean to staff her disaster response team.
5. Procured the submarines and well capping equipment for these depths.
6. Developed a model of the underwater site to make communication about the situation more clear.
7. Authorized the drilling of relief wells for long-term containment.

By opening communication of the situation to the world and inviting engineering help via the Internet, her team encouraged a crowdsourcing and expert sourcing approach to the problem.  As a result, they quickly received estimates on the amount of oil leaking from scientists who were familiar with measuring flows simply based on the video feeds.  Having understood the large magnitude of this flow, the response team was able to garner more dollars to expedite experiments based on simple, back-of-the-napkin estimates of costs due to fines and clean-up that would accumulate each day the well leaked.

Simultaneously, the web site was collecting potential countermeasures from petroleum as well as civil and aeronautical engineers from around the world.  These countermeasures were filtered by the web team and small groups of response team engineers were doing quick research, experiments and models to boil up the most feasible and effective ones. A web-based social media voting and comment system was allowing outside engineers to validate their thinking.  As the most effective countermeasures emerged, the team started to describe experiments necessary to learn how to evaluate the valid sets of potential solutions. Using their growing resources, the response team launched multiple experiments using models and simulations to accelerate their Orient-Observe-Decide-Act loops. Based on what they understood, they took a set-based approach to running these experimental solutions under the sea.

At the end of the first 24-hour cycle, they were clear on the first three underwater efforts.  These efforts were quick, easy and non-destructive to other efforts. Within the next three days, their first experiments did not attempt to slow the leak, but they learn much more about the actual situation of the undersea drill rig, the actual leak size and mix of gas and oil. This data allowed them to update their models and again narrow their choices, as well as feed the root cause and leak containment teams some valuable facts. They were learning and now major equipment was starting to arrive at the site.  They chose to work on the quickest solutions that had the highest estimated effectiveness and least likelihood to ruin the well site for further efforts. All of these models, experiments, and solutions sets were published on the web site in real-time.  The web site formed the basis for governmental and public communication updates as well kept the worldwide crowd of paid and volunteer engineers in the loop.

This learning-first approach led to some quick wins that started to slow the leaks only 10 days after the accident and fully contained it 14 days later.  There was now an estimated 200,000 barrels in the water.  Her attentions turned to other teams. One had the long-term, relief well underway with an estimate of 2 more months to completely contain the band-aided well from other leaks. The results of the response teams efforts kept the total spill size to less than the 250,000 barrels spilled by the Valdez in 1989 and less than the 7 million barrels spilled during Katrina. The Chief Engineer’s teams had used all the best thinking and resources from around the world to narrow to a short-term fixes very quickly.

To conceptually “pay back” the world of volunteers and future deep sea oil teams, the problem sheets, experiment results and retrospective meeting notes are all freely available on the corporate web site.  This site and content are open and shared with the world in an open source manner.  These notes provide data for future Chief Engineering teams to reference during future accidents.  They also provide an engineering case study and market data for equipment suppliers to the petroleum industry to help make these kind of efforts safer in the future. They know that by working fast and leveraging all the world’s resources, they directly attacked the highest economical, ecological and social risk quickly.

Are you a Citizen Engineer?

Things are changing, as we are rightfully blurring the lines between economic, social and environmental responsibility.  Everyone is having to become more responsible to the triple bottom line.  In this new world, the Citizen Engineer needs to be responsible to technology, ecology, society and economics.  In many cases, the Citizen Engineer must acknowledge the difference between problems and difficulties. Problems have answers, but for the difficulties we can do nothing but try to address it in our increasingly large-scale, interconnected and complex world.

Who knows if this approach would lead to a smaller spill in the future, but it would certainly lead to faster learning in the next set of accidents.

How does your engineering team behave during your organization’s accidents?

Ryan Martens is a civil engineer, founding board member of the Entrepreneurs Foundation of Colorado, and CTO at Rally Software Development.

Managing the right goals in software development can have a major impact on your team’s success. You get what you measure. In this regard, the goals you choose can be a defining factor in whether your organization’s Agile adoption thrives or dives. So, what are your choices?

A story about goal-setting

Imagine you are a runner. You have a running coach defining targets for you. Your coach chooses miles and seconds in order to measure you. These metrics turn out to be useful; they are universally understood by you, your coach and your competitors. That’s a good thing. A target goal exchange around these metrics might look like the following:

Coach: “I need you to run a 2-minute mile.”

You: “Uh, I’m not sure I can do that. In fact I don’t think it has ever been done before by any land animal other than a wildebeest, a pronghorn antelope or a cheetah. And, I’m not a wildebeest.”

Coach: “You’re not listening. I already told Sports Illustrated you will. Don’t make me look bad.”

You: “Seriously I’m pretty sure that’s just not humanly possible. I’m no Usain Bolt and I’m not sure even he could sustain a 2-minute mile.”

Coach: “Hey, I set the goals. Run that mile in 2 minutes. Just hunker down and work harder. 2-minutes is the target goal. If you don’t meet it, I’m kicking you off the team.”

You: (muttering under your breath to self, “I can’t do it, you aren’t listening to me, and you are a moron. Why do I even bother to open my mouth?”)

Why goals matter

Get the idea? I’ve been reading John Seddon’s Freedom from Command and Control: Rethinking Management for Lean Service. My colleague Alan Atlas recommended the book. Then my manager Ryan Martens became very interested which led to a book appearing on my desk. Soon after, Karl Scotland mentioned it in an informal conversation about Lean and Kanban. Time to pop it to the top of my reading stack!

Seddon sets a context about Purpose driving Measures that then drive Methods. The combination of purpose and measures start to look like goals. And Seddon breaks goals into two types: Target Goals and Capability Goals. While John’s book is not specifically about software, I could see how we use and abuse goals in our software world. So I decided to delve deeper.

Target goals are arbitrary and doom-laden

Target goals are set to measure hopeful results, often an arbitrary number. Think about our runner story and the 2-minute mile. That target goal was  set by someone else. A target goal typically does not account for your strengths, capabilities , availability and skills. Target goals don’t serve you. They don’t serve organizations either. And yet we cling to them. Consider if, as a developer, I hear the following from my manager:

“Jean! You missed the release deadline we gave you. We told marketing and sales who then told our customers that dev would deliver this specific functionality on this specific date. You’re ruining everything. We’re doomed I tell you, doomed! And all because YOU didn’t hit YOUR deadline.”

Uh, it wasn’t my deadline. And yet, missing that goal is now rippling negatively through the organization. This is a lose-lose situation all due to expectations hinged to an arbitrary target goal.

So why do target goals exist?

In many hierarchical, command-and-control software organizations, target goals are how deadlines are set. As Seddon points out, use of ineffective target goals usually arises due to an impossible purpose coupled with non-value add methods. The measures (or goals) are meant to drive teams to the impossible purpose. A command-and-control organization believes that goals must be set top down. The negative impact of these goals is inextricably woven into unrealistic expectations, death marches, and non-value add methods for controlling the work toward the goals.

Enter the capability goal

Let’s replay the runner scenario again.

Coach: “There is a race coming up, a half marathon. That means we need to start training now. What is your time currently?”

You: “I haven’t been really running lately. Rather than guess my times, I’ll start running now. Do you want to know in terms of how long it takes me to run a mile, or how far I can run in a specific timebox?”

Coach: “Let’s start with how you finish a mile. What is your typical way of getting going?”

You: “First, I do a walk/run combination. Then I can give you my actual mile capability in a week.”

Coach: “Okay, let’s set up a plan and together set some sort of goals based on your capabilities. That will help me guide you and figure out when we are faltering. I know you can do this. I’m going to work with you starting right now.”

You: (muttering under your breath, “This is amazing; yea me! I am going to finally put my passion for running to work, keep improving, and run my first half marathon. I love my coach!)

Capability goals are based on real data

Together, the runner and the coach watch the capabilities of the runner to define goals. The purpose of the goal is clear. Measures are set up that usefully help determine the runner’s progress. The method (or running regimen) for attaining the goal is then applied. Simple.

We use this same approach in Agile teams. A team lead or project manager does not set an arbitrary target goal. Rather, the Product Owner sets an overall purpose/vision. The Agile lead, often referred to as the ScrumMaster, works with the team to assess what goals can be reached based on a team’s capabilities and availability. From iteration to iteration, the team provides feedback on what it is able to complete based on its capabilities. The Product Owner absorbs this information and sets expectations outside the team. The process of continuous improvement continues. The team declares, “Based on what we tend to complete, and given the purpose from the Product Owner, we are going to improve our methods in the following ways.”

When good capability goals go bad

There’s a Gary Larsen “Far Side” cartoon of an open refrigerator with a bowl of glop inside holding a gun. The caption? “When good potato salad goes bad”.

Unfortunately, I’ve seen good capability goals go bad in a number of organizations. Listen to this conversation:

Program Director: “I see that our Agile teams are all now using these things called story points to track their work.”

Me: “Yes, it’s working well. Each team has a good sense of what it is capable of delivering iteration over iteration. That helps me figure out how to help remove impediments, where bottlenecks are, and where any team may need us to better resource them. It’s great!”

Program Director: “Hmmmm.”

Me: (muttering under my breath: “That ‘Hmmmm’ just doesn’t sound good. I think I’m about to get sick.”)

Program Director: “So, what’s the highest story point commitment a team has made?”

Me: “27.”

Program Director: “Great!  I’m setting 27 points as the target goal for all teams to commit to for each iteration. Every point will represent 3 hours of work. This precision will help me calculate exactly what we’ll deliver when. Plus, I’ll know which teams are really working and which ones are slackers.”

Me: “Hmmmm.”

Program Manager: “So, let’s get out there and commit! This Agile stuff is great!”

Me: (muttering under my breath, “We’re doomed, I tell you doomed!”)

What just happened must not be allowed

As you roll out your organizational Agile adoption, you’ll encounter many challenges. Pay attention to the usefulness of capability goals. Insist on using them.  Non-Agile organizations won’t like this. At times, you may feel like Sisyphus pushing a rock up hill for all eternity. Be strong. Don’t allow Agile measurement abuse. Create team and organizational trust.  Remove target goals and embrace the incredible value of capability goals. And, don’t allow any backslide. Don’t let your capability goals devolve slowly into target goals in sheep’s clothing.

You: (muttering under your breath, “You know? This MAY just work. Capability goals versus target goals. For the sake of my team, our organization, and the value we deliver to our customers, I’m going to do it!”)