Engineering Blog

This is the third in a series of posts related to Build Metrics. (Part 1, Part 2) Now that we have a generic framework for sampling build metrics and performing calculations, let’s talk about some of the specific calculations.

In my last post, I mentioned that we have an array named metrics that holds references to javascript objects that perform our calculations. Here’s what it looks like:

var metrics = [ buildsCounted, oldestBuild, jobSuccessRate, timeSinceRed, redToGreen, greenTimePerDay ];

Complex, huh?

As it turns out, the generic algorithm for computing our metrics is pretty straightforward. The particulars of each metric can be a bit complex, although they don’t have to be.

Take this one:

var buildsCounted = {
  name: 'These metrics include:',
  builds: {},
  sample: function(build) {
    var buildName = nameOf(build);
    this.builds[buildName] = (this.builds[buildName]||0) + 1;
  },
  calculate: function() {
    var message = [];
    for(buildName in this.builds) {
      var buildCount = this.builds[buildName];
      message.push('' + buildCount + ' runs of ' + buildName);
    }
    return message.join(", ");
  }
};

Again, this is pretty straightforward. Right? It counts the number of builds of each type. There’s a call in there to a function named nameOf, but that implementation is pretty trivial.

There’s actually more complexity than that.

This metric object relies on a construct similar to a construct that the rest of the metrics leverage frequently – the array named builds. Onto this array, we push the name of a build and the number of builds we’ve counted. Grouping builds by name is useful across the board – here we use it for determining counts of builds by type. Elsewhere, it is useful for other reasons we’ll get to in a bit.

Here’s another metric:

var timeSinceRed = {
  name: 'The light has been green for:',
  builds: {},
  sample: function(build) {
    var defName = nameOf(build);
    if(isPassing(build)) {
      var lastBuild = this.builds[defName];
      if(lastBuild && isPassing(lastBuild)) { return; }
    }
    this.builds[defName] = build;
  },
  calculate: function() {
    if(allPassing(this.builds)) {
      var endTimes = values(this.builds, endTime);
      var lastBuildEnded = Math.max.apply(this, endTimes);
      var diff = new Date() - lastBuildEnded;
      return formatDiff(diff);
    }
    return "Currently Red";
  }
};

Once again, there are several helper methods that operate on either builds or collections of builds to make things easier. The isPassing method checks the build status and the allPassing method examines an object that groups builds by build name (like I talked about earlier) to determine if the most recent build of each type is passing. The function names values plucks an individual property from the most recent build of each type. The formatDiff method takes a time span (in millis) and formats it like “1 hour, 2 minutes, 3 seconds”.

This metric is a little more complicated. The main thing that makes it more complicated is the calculation of when the light went green again after the last red. To refresh, the light is red when any build is failing. It’s considered to have gone green as soon as all builds are passing. The logic in calculate handles that aspect. The logic inside of sample skips recording the build if the last build of that type is already passing.

These are two of the simpler metrics in our system, but hopefully the code samples give you some idea of how more complex metrics could be implemented. The two difficult and interesting metrics are redToGreen, which is the average time between when a light goes red until it goes green again, and greenTimePerDay, which measures the average amount of time per working day (6 AM to 6 PM Boulder time). If there’s enough interest in the comments, I’ll dive into those.