Engineering Blog

An important part of Rally’s culture is to always be getting better. “Continuous improvement” is a principle often associated with various flavors of agile. Regularly holding retrospectives and looking at what’s working and what’s not is a good way to work at it, but how do we know what’s not working?

Here’s my answer: “when was the last time you got kicked in the teeth?”

In various other posts, I’ve talked about getting kicked in the teeth by scalability. Among other things, those kicks told us we needed to get better at memory management, but it wasn’t until we started getting kicked that we started trying to be better at it. Most of the memory issues we found were in old areas of our code and had poor automated test coverage. When we fixed memory leaks, we inevitably broke other things. Kick!

Because we were regularly working with code that was old and hard to test, and because we didn’t want to get kicked again, we got better at end-to-end testing using selenium to drive the browser. How much better? Our browser tests went from around five hours to under thirty minutes despite nearly doubling in quantity. They also became more reliable. Do we still have occasional “flaky” failures? Sure, but between being able to run them ten times as often and by reducing the number of flakies, we’ve significantly improved the value of that particular safety net.

A similar shift has happened in a much less (more?) important area of engineering: the foosball table.

I don’t think it’s a stretch to say that when I started at Rally, I was immediately the best player in engineering. Some people were suddenly receiving uncustomary beatings. It became painful, and they got better. To win, the shots had to get better, the passes crisper, and the possessions more effective. Several people developed go-to shots for the first time.

Do people still screw around on the table? Sure. They just do it better now. There’s ball control going on that makes me shake my head sometimes because I never thought I see it from the players it’s coming from.

A job I had before coming to Rally also taught me a lot about this whole concept. I started there as a contractor and got hired full-time after kicking ass on my first few projects. I kept trying to get better and faster at turning out code. At my one year review, I was shocked and disappointed when I wasn’t promoted to Senior Developer. The reason my boss gave me was a lack of consistent quality. She wanted to be sure I wasn’t a “one hit wonder” before bestowing upon me the superior title.

Looking back, I had hit a few big home runs, but there were plenty of strikeouts too. I was regularly being kicked by my own lack of attention to detail. I started writing more automated tests. I started rehearsing deployments. I automated as much as possible to reduce the possibility that I would make stupid mistakes. All of those things helped. I still kicked myself in the teeth on occasion, but it was infrequent enough that the promotion came.

If you get kicked once, make a mental note. If you keep getting kicked in the same place, do something about it. Not doing anything about it is going to lead to frustration and more pain. The areas that hurt the most are often the areas where we can improve the most.