Steve wrote about our Level One and Level Two experiences with build lights. We’re trying to become third level build nerds, so we wanted to have a place to radiate non-binary (ok, our existing implementation is actually ternary) information about our build. Green and Red (and Yellow) lights help to inform the team what’s going on RIGHT NOW, but they really have no context.

When did the light go green? (I could have asked about red, but I’m an optimist). How long is the light going to be red? How often does the build go red? None of these questions can be easily answered using a three color light.

Rally the app (from now on abbreviated RTA and not to be confused with Rally the company or Rally the engineering team) and Jenkins to the rescue! RTA has some built-in SCM and CI tracking. With the help of a plugin for the CI system of your choice, you can populate RTA with some raw build data every time your CI server completes a build.

After a recent set of meetings aimed at protecting the build light, my team volunteered to help calculate some metrics about our builds. The idea is that by capturing and regularly retro-ing on these metrics, we can measure the impact of team policies and/or engineering practices as they pertain to build quality. We decided that the ideal means of radiating the data is using a custom page within Rally.

Engineering dogfoods RTA. We use it every day, all day. It’s central to how we work. By using a page in RTA for our build metrics, we are creating another information radiator. Our team will see it often, but it has the added benefit of being visible to our salespeople and account managers. We ended up building an app using RTA’s App SDK to track our metrics.

In this post, I’ll touch on some ideas about how we determine build health. In future posts, I’ll explain how we use RTA’s build data and javascript to calculate that information.

There are several types of builds we track. Our “continuous” build runs more than ten thousand tests between javascript and java and is the first thing that gets run after a commit. If the continuous passes, our browser test and flaky finder builds are up next. The browser test build exercises the full stack by driving the browser. It’s pretty easy to write browser tests that are capable of failing intermittently, so the flaky finder build runs any new or changed browser tests 50 times in several threads concurrently to flush out these issues before they compound and cripple the team.

At any given time, our light is green if the most recent build of each type passed. The fact that we track multiple types of builds makes some of our calculations difficult. If you are looking at data, how do you determine when a build went green or went red? What happens in this scenario:

  1. Build A fails
  2. Build B fails
  3. Build A passes
  4. Build B passes

When does the light go red? When build 1 fails. When does the light go green? When build 4 passes. You can’t just check the difference between the end of builds 1 and 3, because the light goes red again in build 2 and Build B doesn’t pass again until build 4.

Confused yet? Checking the state of a light is pretty easy. Applying math to the light is a little harder. Hopefully, this series of posts will be helpful to other teams looking to track the health of their builds.

Request a Call

Looking for support? Open a support case.

Send Us Your Feedback

Provide us with some information about yourself and we'll be in touch soon. * Required Field