Steve wrote about our Level One and Level Two experiences with build lights. We’re trying to become third level build nerds, so we wanted to have a place to radiate non-binary (ok, our existing implementation is actually ternary) information about our build. Green and Red (and Yellow) lights help to inform the team what’s going on RIGHT NOW, but they really have no context.
When did the light go green? (I could have asked about red, but I’m an optimist). How long is the light going to be red? How often does the build go red? None of these questions can be easily answered using a three color light.
Rally the app (from now on abbreviated RTA and not to be confused with Rally the company or Rally the engineering team) and Jenkins to the rescue! RTA has some built-in SCM and CI tracking. With the help of a plugin for the CI system of your choice, you can populate RTA with some raw build data every time your CI server completes a build.
After a recent set of meetings aimed at protecting the build light, my team volunteered to help calculate some metrics about our builds. The idea is that by capturing and regularly retro-ing on these metrics, we can measure the impact of team policies and/or engineering practices as they pertain to build quality. We decided that the ideal means of radiating the data is using a custom page within Rally.
Engineering dogfoods RTA. We use it every day, all day. It’s central to how we work. By using a page in RTA for our build metrics, we are creating another information radiator. Our team will see it often, but it has the added benefit of being visible to our salespeople and account managers. We ended up building an app using RTA’s App SDK to track our metrics.
At any given time, our light is green if the most recent build of each type passed. The fact that we track multiple types of builds makes some of our calculations difficult. If you are looking at data, how do you determine when a build went green or went red? What happens in this scenario:
- Build A fails
- Build B fails
- Build A passes
- Build B passes
When does the light go red? When build 1 fails. When does the light go green? When build 4 passes. You can’t just check the difference between the end of builds 1 and 3, because the light goes red again in build 2 and Build B doesn’t pass again until build 4.
Confused yet? Checking the state of a light is pretty easy. Applying math to the light is a little harder. Hopefully, this series of posts will be helpful to other teams looking to track the health of their builds.