Agile Development Blog

This Friday, February 27th, I will be speaking on a panel at the New Jersey Technology Council’s 2009 CIO Conference called “Moving to a Virtual World” ( www.njtc.org). The panel is on Cloud Computing and it is a mixed collection of vendors and CIO’s talking about the rapid arrival, the clear benefits and struggles of adopting Cloud Computing.

At Rally, we have gained some firsthand knowledge of these technologies, platforms and applications as we try to find the most energy efficient, stable and cost effective solutions for us and our customers.  As a small fast growing technology company, we are an ideal customer for Cloud Computing and I am sitting on this panel as a user, supplier and leader on software development for the cloud.  (We use Amazon EC2, VMWare ESX, Salesforce and seven App-Exchange Apps, Google Apps Premier to run our business and manage our own multi-tenant SaaS/PaaS application in the cloud.)

In preparation for the panel, I was brushing up on some of the latest news and views on this topic.

Here are the worthwhile Cloud Computing links that I used to prepare my talking points:

• Cloud Computing – you have to start on Wikipedia to make sure your definition is grounded
• What is Cloud Computing - a wonderful video of a variety of folks from the Web 2.0 Expo
• Above the Clouds – is a new blog site, whitepaper and video from Cal Berkley Professors on Cloud Computing
• Cloud an Outsourcing Option in Troubled Times – CIO Insight article on using cloud services instead of more traditional IT outsourcing options (both development and operations)
• “What the hell is Cloud Computing?” - Larry Ellison rails on the whole idea as over-blown (see many YouTube videos)

(more…)

Yesterday, I mentioned a Rally client team I visited that was concerned about the Rally tool’s inability to create individual team member burndown charts.  We swam through that torrent of misperception only to enter a shark-infested tank around all of the overhead involved in adopting Scrum. Wait! Isn’t Agile supposed to cut costs, not add them? Big overhead sounds like cost and waste to me. Where is this overhead? We quickly zeroed in on their overhead: the Sprint Backlog. Specifically, what should be tracked in the Sprint Backlog?

To start the conversation, they told me that they are always 100% allocated. Huh? “What do you mean you assume 100% allocation?” They helped me understand that, in the Sprint Backlog, they assumed that they worked 8 dedicated hours everyday. For them, Sprint Backlog should reflect all of that time.

This was followed up with, “So how do we plan for work that we didn’t know would come up, like production issues?“  Clearly, they hadn’t read Slack by Tom DeMarco or heard of the Theory of Constraints and The Goal by Eliyahu Goldratt and the notion of buffer. So, we had a few things to clarify around this “process overhead”:

• What goes in the Sprint Backlog?
• How do you allocate your time effectively?
• How do you “plan” for unplanned work?

In this first post on the Sprint Backlog and using Agile to cut costs, I’ll concentrate on the first bullet:

What goes into the Sprint Backlog?

The Sprint Backlog emerges from the Sprint Planning meeting as the team:

• Identifies the highest priority value items
• Breaks them into tasks
• Determines the effort necessary to complete the value items
• Commits as a team

We cut costs by only committing to and tracking the highest priority items that deliver value in that timebox.

We cut overhead by not tracking items in the Backlog that do not contribute to the potentially shippable product increment.

So back to our overprocessed team. We opened up the Rally tool and moved to their Iteration Backlog view. I saw over 50 items, at least. And then, I understood.

The team was putting EVERY task they do throughout the timebox into the Sprint Backlog, whether it added value to the product increment or not.

Imagine having to track all of your day’s activities. Imagine having to, 2-4 weeks in advance, absolutely commit to these full days’ worth of activities. So, indeed, here was the overhead. Vacation days were in the Sprint Backlog: they had 8 hours of effort. Holidays were in the Sprint Backlog: again 8 hours of effort. The book club had to be tracked separately for each individual who attended it.

The result was that the overall team’s Sprint Backlog would show effort being completed, backlog items being marked “Completed”, the burndown chart burning down nicely, and yet no work of product value was started or in progress. This was not a tool issue; this was an Agile adoption issue.

What is the lesson in all of this?

Be careful about what you choose to track in your Sprint Backlog.

Ensure that what you commit to in that Backlog reflects the delivery of product value.

Abiding by this simple guideline will not only dramatically reduce process overhead, but it will also ensure that what we DO choose to track, whether through a taskboard or an automated tool, is work that is intended to produce product value by the end of the timebox.

Now about their allocation percentages and how it contributes to overhead. That is fodder for another post.

Further Reading:

• Part 2: How Do You Allocate Your Time Effectively?
• Part 3: How Do you Plan for Unplanned Work?

In my last post on Agile Cuts Costs Through Time to Market Improvements, we talked about the 50% savings that are available, but I did not go into why or how Agile teams have the ability to do this.  Time to market improvement in agile project management come conceptually from combining three aspects of agile development. First, the team commits to the discipline of delivering small batches of working, tested increments of “potentially shippable units” of functionality in small time boxes.  Second, the team works with the product owners and customer to sequence those small batch increments into an order that delivers the most valuable features first, in terms of customer value or technical risk reduction.

Investment and payback curve of a software project - Adapted from the book Software-By-Numbers

When a software development team adopts these two principles at acceptable quality levels, beautiful things happen!

1. They shorten the time to feedback and they can now adjust based on customer feedback
2. They open themselves up to shipping early based on delivering value fast
3. They sample faster and increase their ability to react to changes around them

Visually, it would be like shifting the discounted cash (red line) up and to the left. The investment and payback periods become shorter and break even comes sooner.  In addition, because you are delivering working increments sooner with tons of visibility, the risk of the project goes down over time.  (See the picture below.)

Risk only goes down because the time to feedback is shortened and acted on, as shown in the waterfall vs. Agile chart. Factoring the feedback from the visible iteration demonstrations and retrospective into the next round of planning is necessary to enable Agile’s empirical governance model.  This “inspect and adapt” governance approach made famous in Scrum makes agile project management work through active learning and continuous improvement.  This is the critical third pieces of what allows folks to deliver 50% TTM benefits, without it Agile teams can just iterate into a dead-end or black-hole.

With regard to sampling faster, I tend to coach teams to “deliver releases” at least twice as fast as the market or customer requires.  That allows you to get feedback on what was delivered and adjust to changes in the customer environments. (”Deliver releases” does not mean incur all the costs of shipping software, but it does mean create a potentially shippable unit that exercises all your system and integration testing muscles and allows you to get your software into your customer’s hands for trial and feedback.)

Finally with regard to shipping early, Tim Lister of the Atlantic System Guild and Waltzing with Bears/Peopleware fame gave a talk about the potential outcomes of projects at the Agile 2006 conference.  He was talking to his dad about the outcomes of IT projects as on-time or late.  His dad asked him about the other option that apparently no one in IT thinks of – early! Agile teams are always thinking about the option and value of shipping early for feedback.

The great thing about TTM savings is that these savings are mostly independant of productivity savings that I mentioned in an early blog post.  Next post is how Agile Cuts Cost by building the “right” software that actually get’s used.  Finally, I will talk about how these savings and behavior changes play together to deliver astonshing results over time.

In this post, I’d like to comment on a request from a Rally client I recently visited. Their tool request? A burndown chart for each individual. We were in a small meeting and my intent had been to help them with their Agile adoption. “No, Jean, it is your Rally tool that is the problem.” Hmmmm. Let’s take a look. What seems to be the problem?  “Your tool is not helping our ScrumMaster and management team track individual team members’ burndown during the Sprint. We absolutely have to have that.” Huh?  I had to call foul and get the group to step back and talk about “What’s in a burndown chart and why?”

Individual team member tracking in a burndown chart is a bad smell. It suggests that the commitment made in the Sprint Planning meeting is not a team commitment. It also suggests that the management is not tracking value delivery, but rather team work. I dug further with this group about that. Each individual at this organization was expected to commit exactly to the number of hours any of their tasks would take, and stick to it. That commitment was meant to never change. If a team member was “taking too long” (which they claimed could only be known via individual burndown charts), management would then be able to “do something” about it. What happened to the Daily Standup for team check-in? But I digress.

Tracking in a burndown chart is meant to trace team commitment around the valued items it intends to deliver. At the end of the Sprint Planning meeting, it reflects the team’s initial estimate of effort to achieve that value delivery. During the Sprint,  the burndown chart reflects daily, “Are we going to meet our commitment to delivering this value? If not, what is the single most important, the highest priority thing we must do today?” Finally, the burndown chart  informs the Sprint Demo and Review about what happened with the team’s commitments to value delivery  versus what they were able to complete.

What’s in a name? Well, when it is a burndown chart, there are three things:

1. charting the initial Sprint Planning meeting team-wide commitment to the valued items the team can deliver in a timebox
2. prompting the daily inspection of what is the most important thing the team must do today to meet that commitment
3. informing the the team at Sprint review to evaluate how the team’s estimates and commitment compared to what value the team was able to deliver

About the Author: Jean Tabaka is a wine enthusiast, author and Agile Fellow at Rally Software Development. Subscribe today to get free updates by email or RSS.

In the mid-1990’s, I worked on a great team (which included three members of Rally’s current technical team) as a consultant and eventually as a Director of IT at US WEST Communications in a part of the IT organization that became known at the Global Village Labs.

You can read about some of our early Intranet work in this 1995 Fortune Magazine article.  In addition to being filled with great people, we had a great leader by the name of Sherman Woo (great profile on ZoomInfo), who was a complete rebel.

Sherman, a 25-year veteran of the Bell system, was rebelling against slow IT services in the age of the 1996 Telecommunications deregulation act. To meet the rapidly changing needs of open network, he built this amazingly agile organization that leveraged Mosaic and then Netscape browsers to screen scrape mainframes using ORAPerl.

We used these agile teams of 5 to 10 people to build tools for customer service representatives and field technicians.  These were typically simple tools that bridged a couple of mainframes to build compound views that could answer really tough questions.

These questions were hard because each role was only trained in a limited scope in limited mainframes and that full training took 24 months.  So we used early Internet technology to open the access and bridge that gap with simple tools.

We made it into Fortune Magazine because our time to market was measured in months and our ROI was 1000%.  In our agile model, we delivered our first working increment to the business on test servers in a fixed time box of six weeks.  If these solutions showed value and promise for 5 users, we would run another 6-week time box and then roll the application to 50 users.  Successful projects then went on to 500, 5,000 and then up to 20,000+ users following this 6-week time box approach.  Marginally valuable applications got tabled at the end of the 5 or 50 user-demonstration time-box.

As a result, we grew the group from the initial 10 folks to the 150 person organization in IT in just 18 months.  (See the internal brochure we used to drive business to our group – (BTW, this was really fun to research on the Internet – such good memories.)

QSMA Agile Impact Report showing Agile teams 25 to 50% faster versus 7,500 other projects

50% Faster Time to Market OR 50% Less Staff?

It was all about Time to Market (TTM) for GV Labs and for most folks adopting Agile in this decade.  However, now with the cost-cutting focus of these turbulent times, we see many organizations trading off some of the TTM for cost savings.  As a result, they will use fewer Agile teams on a project and deliver a lower throughput per agile, time-box cycle.  In many cases they are using fewer people, because they have fewer people due to staff cuts.  (If you have not taken staff cuts and are wondering how to do that and maximize your Agile success, read an earlier article of mine on TechTarget – Cutting Your Way to Increased Agility and my post on Israel Gat’s Social Contract for Agile Software Development)

Of course this is a continuum of speed and these savings exist in either getting done faster with more or getting done slower with less.  You and your organization should be careful to consider the total cost and benefit of the project when right-sizing the team size to completion date. There is an optimal range for most projects and with true agile project teams you get the benefit of adjusting this based on actual performance over short time-boxes.

To gain these benefits, you have to actually become Agile and not just adopt some of the practices. Agile means dedicating you and your team to learning and continuous improvement.  We did it at US WEST in the 90’s and QSMA shows how many of our customers are doing it today.

If you truely become agile in these times, you can accelerate out of this downtrun smarter, stronger and leaner.

Further Reading:

• Brutal Prioritization in Agile: cut costs by NOT building the fluff
• Agile Cuts Costs by a  Factor of Four – or More!
• How Does Agile Deliver Time to Market Improvements of 50%?

The various sizes of and shapes of Ecospheres

As you may know, I am very passionate about the concepts of sustainability.  For my birthday this year, my thoughtful wife and son gave me a living example of sustainability that sits on my desk and reminds me to work on this topic everyday.  It is called an EcoSphere and I got it from EcoProducts Home Store here in Boulder (they are folks who also provide all of our compostables that help us approach zero waste here at Rally – they are a great Boulder success story).

The Ecosphere is a completely closed world and is self-sustainable. It was developed by two NASA scientists (the late Dr. Joe Hanson and the late Dr. Clair Folsome) trying to create models for long-term space flight.  In it you will find algae, shrimp and bacteria living in a closed cycle with sun light as energy. (Learn more about EcoSphere’s sustainable model on their site.)

At $125 for a small sphere they are a bit pricey, but they are fun to watch, people always notice the shrimp moving around on my desk.  After attending Thomas Friedman’s talk on “Hot, Flat and Crowded,” I realize they are just another part of my “work in progress” to living sustainably in the U.S.  These do not feed me like my chickens and goats, but they also do not eat like my dogs and cats.

Here is mine on my desk - cool eh?

Definitely the most common question is:  “Are those sea monkeys?“  According to the web site, they live 2 to 7 years with really no maintenance and NO they are not sea monkey branded brine shrimp.

This EcoSphere keeps me focused on the long road of continuous improvement needed to make our industry a zero carbon footprint or sustainable industry.  We are currently on the road to be a larger emitter of CO2 than the airline industry by 2020.

We have to find the innovations in infrastructure, the methods and tools to reverse this.  My of view of how to reverse this behavior is through the emerging software value cycle that is made possible by SaaS/Clouds, Agile development and Web 2.0 customer communities.  You can read my thoughts on these topics or hear a MassTech webinar.

I have about 15 of these guys in my Eco-sphere.

I believe that with the change to Lean thinking – from products to services and with virtual connections to customers – we can learn to quickly adapt and adopt to new sustainable products and behaviors.

We need a value chain in the IT industry that is closed loop and sustainable, not open loop like the Story of Stuff.

I encourage you to take a moment and consider getting one of these model worlds for yourself or your best friends.  It will keep you on the road to smarter, leaner and greener.

About the Author: Ryan Martens is an avid outdoorsman, founding board member of the EFCO, and Founder and CTO at Rally Software Development. Subscribe today to get free updates by email or RSS.

Further Reading:

• Let’s stop splitting hairs – We need to get to zero carbon output to really make a difference
• Leadership in Sustainability – Blog Talk Radio Interview

In this 20-minute interview with Michael Vizard, I do a wide survey of why Agile development practices can be so effective at cutting the costs of development and operation of strategic IT applications. Michael asked some great questions, including whether the current economic recession will stimulate change in the development process or cause it to stagnate, how people will train and staff for the coming changes, and how today’s ALM tools differ from what he refers to as the “old guard,” who are starting to claim that their tools also support Agile development.

Read related posts on:

Can you afford the software you are developing? (from Israel Gat’s blog)

Cutting costs through productivity improvements, and

WHY do you develop and operate software?

Let me know what you think!

Back in 2007, Jean and I developed an evolutionary model of Agile adoption for teams and organizations seeking the benefits of “scaling software agility.”  We presented talks on this at the Agile 2007 and Agile 2008 conferences.  We call it Flow-Pull-Innovate based on the Toyota Lean principles of Flow-Pull-Perfect.

Productivity gains in Agile development teams come conceptually from eliminating waste.  Just getting to the first step of Agile maturity can lead to 10 to 20% productivity increases.  We want you to be successful taking the first step, and there is a ton of opportunity in most software development shops. According to Tom and Mary Poppendieck, most software development organizations are only spending 6% of their time doing value-added tasks.  The other time is wasted in these categories:

• Partially done work (coding features that get removed from release)
• Extra processes (elaborating requirements that do not get built)
• Extra features (features that are rarely used – more on this in another post)
• Task switching (across multiple projects – losing flow)
• Waiting (requirements, designs, feedback, builds, other teams, larger organization, customer)
• Motion (walking over to interrupt folks for a build or status)
• Defect (internal or external)

Agile works to systematically address these wastes as you mature.  In our white paper on moving to Program Pull, Jean and I characterize the steps, roadblocks and benefits found at each step in the Flow-Pull-Innovate maturation process.

On the move to Flow, we find that teams can increase their productivity working as a dedicated team on a single project for a two-week iteration cycle.  The reduction in waiting, motion, task switching and partially done work can be dramatic based on your level of multi-tasking.  We even find some teams in Flow reduce their extra features, partially done work and defects, but those productivity gains tend to be more associated with the move to Agile Pull.

According to the QSM Associates study of Agile development teams, you can expect to save 16% (the green dots on the graph). Of course the teams that worked with Rally’s products and services achieved on average 25% productivity gains (the red dots on the graph).

(The graph on the right from QSMA shows a white line of the average productivity of different sized projects in their 7,500+ project database. Between the blue dotted lines is the area 1 standard deviation from the norm.  You can see only one Agile project was less productive that the average and  7 Agile projects of the total 29 were outside 1 standard deviation. Thus 1/4 of these Agile projects were in the top 16% of all projects ever benchmarked by QSMA – see Michael Mah’s blog for more discussion – OptimalFriction)

Of course,  you cannot have your cake and eat it at the same time.  That is, you cannot just start running two-week iterations and become 20% more productive overnight, but with a combination of services and Agile project management solutions, you can attain these results in months. That is why we recommend a two-pronged approach to cutting costs with Agile and Rally offers a Guarenteed Succes program that combines our world class services and application in a discounted bundle for new and existing customers.  Get started by getting your teams to Flow and the savings can come fast.

Yesterday evening, I was listening to “Marketplace” on National Public Radio and a segment on “Choosing the Best Business Books”.

Kai Ryssdal, interviewing Jack Covert and Todd Sattersten, asked the two if there is one book they’d recommend to survive a downturn economy. The two have just authored The 100 Best Business Books of All Time so they seemed like the right people to ask.  Their answer surprised me in a particularly wonderful way. SPOILER: it wasn’t my book Collaboration Explained :-)

But before I tell you which book it was, let me tell you a little about what made me so happy and amazed.

I teach a class called “Collaboration Explained” through Rally’s Agile University. In this class about facilitating Agile software teams to be truly collaborative, I also touch into how to tell when I team really isn’t collaborative. It could be about the project’s or organization’s leadership; it could be a fundamentally toxic environment.

Whatever the cause, I talk about (1) recognizing certain dysfunctions, (2) why we should care about them in Agile software development, and (3) what to do about them.

This brings us back to the book recommended by Covert and Sattersten. Both my source on Agile team dysfunctions and the business book they recommended for this difficult economy are the same: The Five Dysfunctions of a Team: A Leadership Fable by Patrick M. Lencioni.

What? A book about teams and their dysfunctions as THE business book for surviving a downturn economy? Yes.

Teams in dysfunction can’t perform. And teams that can’t perform can’t be productive. Non-productive teams eat away at organizational performance.

Think about a team you have been on that just never seemed to gel. Stuck in what is sometimes called “Storming”, it just never seemed to be able to pull it together.

Thinking about that team, do some of these descriptions sound familiar?

• Absence of trust
• Fear of conflict
• Lack of commitment
• Avoidance of accountability
• Inattention to results

As Lencioni in his fable will tell you, these dysfunctions build on one another; that is, until you resolve a fundamental absence of trust, you won’t be able to tackle a fear of conflict. And so on.  Individuals on teams stuck in these dysfunctions  “act out” in ways that evidence just how pernicious the damage can be from these untreated team ailments:

• Invulnerability
• Artificial harmony
• Ambiguity
• Low Standards
• Status and Ego

So, back to “Marketplace”, Kai Ryssdal, Jack Covert and Todd Sattersten. Covert and Sattersten believe in the power of this business fable during this downturn economy because of the power and importance of teams in bringing economies back to life. If you are looking to Agile and how it can cut costs for your organization, consider the power of your teams. Work to engender trust and promote an ability to have constructive conflict. Empower your teams and amplify the learning that teams bring to organizations. And, consider that THE book recommended for all businesses by Covert and Sattersten, not just for software teams, in this downturn economy is a book that directly addresses team dysfunctions.

Cut costs, yes. But don’t just cut groups without consideration of Lencioni’s advice. Work to move dysfunctional teams to becoming high-performing. Maintain and nurture those teams that continue to gel and perform. That is how to survive this economy.

Further Reading:

• Lean economic times call for Lean, Agile software development
• Brutal Prioritization in Agile: cut costs by NOT building the fluff stuff
  

In December, our friends Jim Duggan and Thomas Murphy at Gartner completed a long-anticipated report on what they call the Application Lifecycle Management (ALM) space.  This space has been primarily dominated by source code management, integrated development, bug tracking and testing vendors.

With the rise of Agile development and growth of distributed development, the ALM space has expanded to include development management solutions like Rally.

This report is what Gartner calls the  “ALM Marketscope” and it is a broad evaluation of the ALM market and the providers within it.

There’s good news in the report from both a Rally/vendor standpoint and a customer standpoint.

Gartner estimates that sales of ALM tools totaled $1.26 billion for calendar year 2007, with a growth rate of 11.2%. With regards to the current recession, the report says, “Although current economic conditions will suppress many projects, we believe that the ALM market will stay relatively strong because of the value that ALM returns to a company in productivity, predictability, automation and governance.“

That’s good news for ALM providers like us for obvious reasons, but it’s also good news for users because it shows that real results are being achieved with the applications of ALM solutions in all sizes of software development organizations.

This next generation of ALM tools and applications are enabling better performance in software development even while software development is becoming more central to delivering business value, more distributed and more complex.  These advances are made possible through the use of new team collaboration solutions that are increasing visibility and transparency across the entire software development lifecycle.

We were admittedly also very happy to receive a “positive” rating in the report.

Here are a couple of quick hits about Rally
for the full report,  go to Gartner’s web site.

• Rally “has the strongest overall support for Agile.”

• “It promotes the use of ‘lean’ principles across the ALM life cycle and offers a strong, independent toolset that integrates with many products through a solid architecture. The product supports requirements, test case, defect, program, project and product management functions.”

• “It recognizes the need for more than project management and that more roles are involved than developers or project managers. The offering is flexible and well supported by Rally’s training and consulting offerings, including an online community and the Agile University.”

• “Rally has invested heavily in scaling to mirror very large and complex, multiteam organizational structures. They have deployments with more than 1,000 seats.”

Obviously, I am thrilled that we received such positive comments in the survey.  For a company that is only five years old, it is very satisfying to be ranked second only to IBM in this space.  Obviously this was a huge team effort to build Rally, but a special thanks goes to all of our customers who have engaged with us to shape our Agile lifecycle management platform over the last four years and 31 product releases on eight-week cycles.

It’s good to be second – we will keep trying harder!  Let us know what we can do to be better by commenting on this post.

About the Author: Ryan Martens is an avid outdoorsman, founding board member of the EFCO, and Founder and CTO at Rally Software Development. Subscribe today to get free updates by email or RSS.