Engineering Blog

There have been a number of blog posts and discussions lately around the evils of agile. Lots of traffic recently surrounding an old post by Cedric. This stackoverflowist is switching to waterfall. Hell, agile ruined this guy’s life – HIS ENTIRE LIFE!

I’m a big fan of agile software development, which should come as no surprise given that I work at Rally. But honestly, I understand where all of this backlash is coming from. Whenever someone has come to me and told me “this is the way you will do things now,” then decreed some elaborate, formal process I would have to follow, I’ve resented it. Don’t tell me how to work.

But I think a lot of people who are annoyed at agile lately are confusing Agile with agile.

(more…)

At Rally, we’ve stayed committed for the last 7 years to never telling the business that the product has to halt active development to pay down technical debt. For us, the “big rewrite in the sky” has always been off the table. Instead, we prefer to incrementally refactor and improve the existing elements of the codebase, gradually getting it to where we want it to be without ever halting feature development completely.

This has never been an easy task. The truth is, the codebase is quite old by engineering standards, many parts of it written before standards emerged in the industry. Our entire persistence layer, for example, was created by hand well before libraries such as Hibernate existed (or at least, before they became standard practice). This means that there are many, many areas of the code that we’d like to attack and improve.

One thing we’ve started doing is holding lunchtime “refactotums,” where we take a look at some code and try to improve it. Of course, with such a large codebase, and so many areas of the code that have not been touched since they were first written, we’ve often wondered how to most effectively spend our refactotum time. This led us to wonder “what are the areas of the code where a refactoring would have the greatest impact?”

We started keeping track of our codebase using Sonar which was extremely helpful in keeping track of metrics at a class level, but it didn’t give us quite enough information. For example, if Class A has a complexity score of 500 and Class B has a complexity score of 200 (higher is worse), you’d think that spending some time cleaning up Class A would be the most effective thing to do. But what if Class B is used by 50 times as many other classes as Class A? Or what if Class B, complicated as it is, has never been modified in the history of the project? It would be nice to clean this class up, but it might be a better use of time to focus on Class A

So our idea was to combine two metrics: the class complexity (based on McCabe Cyclomatic Complexity) and the number of times a file has been modified in the codebase. The idea here was, every time a file has to be modified, it has to be read. A complex file is harder to read, making the engineer more likely to misunderstand or make a mistake while editing the file.

Luckily, git makes it quite easy to count the number of modifications to files in the codebase:

(more…)

One of the most common sources of tension between product owners and developers is when product owners are surprised at how high an estimate for a story might be. Usually this tension is easy to resolve by reiterating that the product owners really have no concept of how much something should cost. However, there is one scenario I see over and over again: when a product owner protests the estimate of a story because it seems, to the PO, like it’s simply re-using an aspect of the system somewhere else.

“I already paid for that!” he or she might say, quite correctly. A simple example recently for us here was our column picker. Our grids allow you to select which columns you wish to see on any given grid. Later on, we created the ability to have grids inside of panels on user dashboards. Our product owner wanted to put the column picker on the settings pane of a grid panel. To him, this was simply reusing the column picker that had already been created for an earlier story. He was quite shocked when our estimate for the effort was just as high as it was the first time.

His frustration is completely understandable. How is this possible? Why isn’t it easy to just drop that component into another place and have it work? After all, if we were good engineers, wouldn’t we have written that component to be reusable?

Actually, no. It’s because we were good engineers that it wasn’t written in that way. This is one of the hardest things for product owners to understand, and I hope to explain it in this post.

Evolutionary Design

Once upon a time, development teams worked in specific phases. First they would design all of the code they were going to write; boxes would be drawn on whiteboards, sequences captured in documents, and so on. Once the design was finished, the developers would go off and write the code for it. Luckily, our industry has realized that this is a problematic way to write software.

It’s simply not likely that the team will discover all of the quirks of the design during the design phase. It’s even more likely that requirements and features will change during the long span of time this process takes, forcing the developers to throw away work and go “back to the drawing board” on the design. Splitting the design phase from the coding phase is a recipe for disaster.

Instead, it’s generally preferable to practice evolutionary design; allow the design to grow organically as the code is written. This, combined with testing and refactoring, seems to be the best known way to ensure that the quality of the code remains high as a product grows.

Doing this makes it easier to deal with changes that effect design, and it prevents the team from wasting time going “back to the drawing board” in the event a problem is discovered.

YAGNI

One principle to follow as you code is the YAGNI principle. YAGNI means “You Ain’t Gonna Need It” and essentially it means that developers should discourage themselves from adding additional code to support something that is not, at the time, required. Developers have a strong desire to add code here or there while editing code, and YAGNI helps us remember to resist the urge to add superfluous code or functionality.

Following this principle helps eliminate waste, and helps keep developers focused on adding value that product owners want. But why?

If I’m editing Module A to add Functionality X, why not just add Functionality Y while I’m there? It may delay the delivering of Functionality X in the short-term, but later when it’s decided that Functionality Y must be supported as well, it will be dirt cheap, right?

The trick to understanding why YAGNI helps is realizing that there are thousands upon thousands of potential Module As in a system. The truth is, Functionality Y may simply never be required. And even if it is, Functionality Z may not be. Adding that functionality would be a waste of time and resources if it’s never used.

YAGNI Applied to Design

The same principle behind YAGNI can be extended to aspects of high-level design. When we originally designed the Column Picker component, it was not designed to be dropped into any part of the system. This was intentional: if we had spent the time to make the Column Picker completely generic and reusable, but then had never needed to drop it into another part of the system, the additional effort required to make it super-generic would have been a wasted effort. This is true for any aspect of the system being designed: we can spend the extra time and effort to make something extremely generic, but if that generality is never needed, that time is waste. Many times developers will refer to something that’s overly generic as “over-engineered.”

This means that the description of the component can often be misleading. We never built “a column picker” – we built “a column picker for a grid page”. The column picker was written in a way that assumed it was part of a grid page. Again, it could have been written more generically, but doing so would have potentially created waste (this does not mean that it’s alright to design code that violates principles of good design. Code can be well-designed but still not fully generic).

So when it came time to put the component in a completely different context than the one for which it was designed, it was still a lot of work: almost as much as was originally required to make the first pass. Though some time was saved simply because it was a known and well-understood problem for which we would leverage newly-developed knowledge, it still effectively required writing a whole new component: one that was much more generic and utilized what it could of the previous increment.

Essentially, to avoid having developers waste time, components should be “generic on demand”. Once a component needs to be reused, then it should be refactored into something extremely generic, but not before. The end result is that a product owner will have to pay for a component twice before it’s cheap.

This may seem counterintuitive to product owners, but it’s important to understand. Because the team approaches coding in a way that causes the second time to be nearly as expensive as the first, the team is able to deliver everything else quickly.

The first time, it is costly because the developers have to solve unknowns and create something new from scratch.

The second time, it is costly because it has to be molded into something generic and reusable.

The third time, however, is cheap.

Many developers think of unit tests as tests that test a single class. In fact, I myself once thought this way. If I wanted to write unit tests for a two-class system in which a class used another class, I’d write two unit tests. After all, if I created instances of both classes in my test, that wouldn’t really be a unit test, would it?

In recent years, I have revised my stance on this matter. This distinction between what is and is not a unit test is one I no longer draw in the same way.

Unit Tests need to be:

• Fast – Unit Tests should never run so slowly that they discourage developers from running them all the time.
• Focused – Unit Tests should focus on a single area so that they can isolate problems to that area when they fail.

Once upon a time, I thought ‘focused’ meant that it should be focused on a single class. Instead, it should focus on an area, but that area can include multiple classes.

(more…)

Many of us at Rally like to Pair-Program because it offers continuous code reviews and many other benefits. We also like using Git, because it’s fast and makes branching and merging far easier than Subversion.

Unfortunately, pair programming can often make interacting with a version control system difficult. Who officially makes the commit? How do you make sure the other half of the pair gets credit/blame when needed?

Luckily, Bryan Helmkamp wrote a script that allows pairs to update the usernames of commits to reflect the involvement of both people.

We’ve been using this script, but we’ve made some changes and figured we’d let others use the modified script as well.

(more…)

Everyone knows the drill with Agile – the developers put estimates on stories, and those estimates are used to plan releases. Of course, some releases go awry, with the engineering team delivering far fewer completed stories than expected. When this happens, people start to ask “why are our estimates so inaccurate?” What many don’t realize, however, is that this is the wrong question. Estimates don’t need to be accurate, they need to be consistent.

Our velocity corrects for the inaccuracy of our estimates. As long as engineers are consistently wrong, the velocity will ensure that releases can be planned and executed smoothly. If releases start to become increasingly bumpy and unpredictable over time, there’s a good chance that your consistency is to blame. Poor consistency means rocky, less predictable releases, which in turn can lead to incurring technical debt in order to meet release commitments.

Here are five things to watch out for within your team. These are inhibitors to consistency that typically will plague a seasoned agile team, not a new team transitioning into agile.

(more…)