Engineering Blog

One of the things I really enjoy about being in Operations is the variety of work that comes your way. Some of this variety comes from projects in different areas of the system, but some of this variety comes from being exposed to interrupts that come along each day. While Agile methods are designed to allow for pivots, handling daily interrupts on the scale that you see in Operations can be very intrusive to the process.

Most organizations have an on-call schedule to handle after-hours issues on a rotating basis, but we have taken that one step further. What we wanted to do was try to offload much of that daily interrupt driven work away from those folks working through their normal project oriented stories. This has the effect of allowing the majority of the team to focus for 3 weeks at a time (we have a 4 person rotation) while one person handles the majority of these interrupt type issues. This lucky person is the “Op of the Week” or OW for short.

A similar idea exists on the Development side, we call them the “Dev of the Week” or DW. They are the first point of escalation for the OW on application related issues where we need Dev team assistance.

One of the areas we have struggled a little bit with our OW rotation is making sure everyone agrees about what falls under the responsibility of the OW. As a result we’ve created a document which defines our working agreements around the OW role. We review this as a team and make adjustments where necessary.

Here’s an overview of the types of stuff we focus the OW on for their given week & the role we expect them to hold:

Rotation: OW duties rotate every Monday at 12pm. This provides time for the outgoing OW to provide an update at SoS (Scrum of Scrums) and to provide any turnover to the new OW.

General Duties:

• Respond to all critical pages (Failed hardware, software, high load events, etc.).
• Communicate status to team when issues are being worked. Include DW & general Dev team as appropriate.
• File defects when needed for application events or problems that require assistance from DW or Dev team.
• Issues raised during OW are owned to completion, coordinating additional resources as necessary.
• Attend Scrum of Scrums & provide status on any new Ops issues (customer problems, production outages, etc)
• Respond to all incoming operations interrupt type requests (includes all Ops responsibilities, not just prod)
• Remediation of critical security issues. Drive required security patches / workarounds to completion.
• Review nightly jobs including Backups for errors / problems.

We also maintain a calendar of events in this same document which describes the routine daily activities which are part of the OW duties. For example, because we perform weekly deploys to production we have a pre-production push on Thursday evening of the Release Candidate that the OW performs. Saturday mornings when our production release rolls out, the OW is expected to be on high alert for problems.

This process has worked well with some adjustment over time. It is also not intended to be completely rigid. For example, we certainly have cases where something will come up around a certain part of our infrastructure where the OW is not the expert. When appropriate, and where both parties agree, other team members are welcome to take ownership of issues that come up if they are passionate about getting those issues resolved and/or have expertise in that area. We don’t expect team members to do this all the time though, as being OW also forces some cross-training, something we think is really important.

If you’ve found a good way to offload interrupts or have feedback on what we’ve done, let us know in the comments.

Today we want to share a nice integration between JsTestDriver and SinonJS that allows for great stubbing and mocking facilities in your JavaScript tests.

A common thing needed when writing JavaScript unit tests is to mock out an object or the return value of a function. In addition to this, it’s important to restore the original object or function at the end of the test. Let’s check out a pretty trivial example:

Rally.getVersion = function() {
  return 2;
};

Rally.isSupported = function(operation) {
  return Rally.getVersion() > 1;
};

TestCase('IsSupportedTest', {
  'test should support version greater than one': function() {
    assertTrue(Rally.isSupported('read'));
  },
  'test should not support version less than two': function() {
    var orig = Rally.getVersion = function() { return 1; };
    try {
      assertFalse(Rally.isSupported('read'));
    } finally {
      Rally.getVersion = orig;
    }
  }
});

Now let’s say you have 20 tests around the Rally.isSupported method. Each of which you might need to clean up any mocked out methods. This would get tedious, so you might leverage the setUp and tearDown methods provided by JsTestDriver’s TestCase:

TestCase('IsSupportedTest', {
  setUp: function() {
    this.origGetVersion = Rally.getVersion;
  },
  tearDown: function() {
    Rally.getVersion = this.origGetVersion;
  },

  'test should support version greater than one': function() {
    assertTrue(Rally.isSupported('read'));
  },
  'test should not support version less than two': function() {
    Rally.getVersion = function() { return 1; };
    assertFalse(Rally.isSupported('read'));
  }
});

Another way to do this would be to use the SinonJS framework, specifically the stub() method. This allows you to stub out an existing method on an object, and provide expected return values as well as a variety of methods to assert the method was called:

TestCase('IsSupportedTest', {
  setUp: function() {
    this.getVersion = sinon.stub(Rally, 'getVersion');
  },
  tearDown: function() {
    this.getVersion.restore();
  },

  'test should support version greater than one': function() {
    this.getVersion.returns(2);
    assertTrue(Rally.isSupported('read'));
  },
  'test should not support version less than two': function() {
    this.getVersion.returns(1);
    assertFalse(Rally.isSupported('read'));
  }
});

This is nice and clean. The restore() method will automatically unwrap the stubbed method, and we can easily control the return values of the stubbed method. But we can make this even cleaner. We can have sinon automatically restore all stubbed and mocked methods upon destruction of the TestCase object. This can be done by leveraging Sinon’s sinon.testCase, which is an integration wrapper between SinonJS and JsTestDriver.

(function() {
    var origJstestDriverTestCase = window.TestCase;
    var origSinonTestCase = window.sinon.testCase;

    window.sinon.testCase = function() {
        throw Error('** Do not call sinon.testCase() directly, TestCase() will do this for you! **');
    };

    var globalSetUp = function() {
        // ... global setup stuff goes here
        if(this.originalSetUp) {
            this.originalSetUp();
        }
    };

    var globalTearDown = function() {
        // ... global tear down stuff goes here
        if(this.originalTearDown) {
            this.originalTearDown();
        }
    };

    window.TestCase = function(name, prototype, optType) {
        prototype.originalSetUp = prototype.setUp;
        prototype.setUp = globalSetUp;

        prototype.originalTearDown = prototype.tearDown;
        prototype.tearDown = globalTearDown;

        prototype = origSinonTestCase(prototype);
        return origJstestDriverTestCase(name, prototype, optType);
    };
})();

With this in place, we can now write the following test:

TestCase('IsSupportedTest', {
  setUp: function() {
    this.getVersion = this.stub(Rally, 'getVersion');
  },

  'test should support version greater than one': function() {
    this.getVersion.returns(2);
    assertTrue(Rally.isSupported('read'));
  },
  'test should not support version less than two': function() {
    this.getVersion.returns(1);
    assertFalse(Rally.isSupported('read'));
  }
});

Wrapping the TestCase’s prototype with sinon.testCase exposes all of the SinonJS methods through the TestCase object – so this.stub(), this.mock(), etc will all work. Additionally, SinonJS will automatically unwrap the stubbed/mocked objects at the end of the test. We’ve also put some logic in to ensure the following will throw an error:

TestCase('IsSupportedTest', sinon.testCase({  // <--- don't need to do this, it's done automatically
  setUp: function() {
    this.getVersion = this.stub(Rally, 'getVersion');
  },

  'test should support version greater than one': function() {
    this.getVersion.returns(2);
    assertTrue(Rally.isSupported('read'));
  },
  'test should not support version less than two': function() {
    this.getVersion.returns(1);
    assertFalse(Rally.isSupported('read'));
  }
}));

In my second post I explained how to get started making your own Apps using Rally’s APP SDK. In this post I will show you how to use the starter kit to create an App that shows the stories that have changed in the past day.

First we will use the starter kit to create up a new App.

Note: In the example image you will see that I had to escape the brackets in order to get the rake task to run in oh-my-zsh.

Open the LastDay.js file that is contained in the folder that the starter kit created for you. For editing JavaScript I like the JetBrains IDEs, for this tutorial I will be using RubyMine to help create the App.

One of the important things to remember when using the SDK is to make sure that all of your JavaScript code is called after the addOnLoad callback has been completed. The addOnLoad callback is called by the underlying SDK code after all the resources that are needed are loaded and the environment for your App has been set up.

For this App we will use the RallyDataSource to query Rally for work items that have been changed in the last day and put those items into a table.

First we need to configure the RallyDataSource. By default a RallyDataSource will follow the current projects scoping. The server replaces the special variables like __PROJECT_OID__ with the current scoping information from your Rally session. When you change scope in Rally your App will be reloaded and those scoping variables will be replaced with the updated project information.

var rallyDataSource = new rally.sdk.data.RallyDataSource("__WORKSPACE_OID__",
"__PROJECT_OID__",
"__PROJECT_SCOPING_UP__",
"__PROJECT_SCOPING_DOWN__");

Since we are going to only show stories that have changed in the last day we will need to calculate yesterday’s date.

var yesterday = rally.sdk.util.DateTime.add(new Date(), "day", -1);
var startOfYesterday = new Date(yesterday.getFullYear(), yesterday.getMonth(), yesterday.getDate());
var stringDate = rally.sdk.util.DateTime.toIsoString(startOfYesterday);

The SDK ships with a some utilities to make it easier to work with dates. In the above code I find the DateTime for today and subtract one day from it. I then create a new date with the year, month, and day to get a DateTime that is at the beginning of the day. Lastly that DateTime is translated into an ISO string (the format that Rally expects dates to be when they are sent.)

Next we need to create the configuration object for our table.

var config = {
//this is the type for User Story
type: "hierarchicalrequirement",
query: "( LastUpdateDate > " + stringDate + " )",
columns:
[
{key: 'FormattedID', header: 'Formatted ID', width: 100},
{key: 'Name'},
{key: 'ScheduleState', header: 'Schedule State'}
]
};

• type: Type is the Rally item type that you want the table to query for. In this case we want to display User Stories so we use the name WSAPI type name for them which is “hierarchicalrequirement”. You will notice that the type name has all spaces removed from it so if you wanted to see last changed defect suites the type would be “defectsuite”
• query: Query is a filter you want to put on the data that will shown in your table. LastUpdateDate is changed every time that an artifact is changed in Rally. In the case of our query we want things that have a LastUpdateDate greater than yesterday.
• columns: Columns is an array that contains what fields you want displayed in your table.

Now we need a place to display the table. First we need to create a div to show it in. That can be done by opening up the LastDay.template.html file and putting a div tag into the body.

<body class="lastDay">
<div id="myDiv">

</div>
</body>

While we are in here we can change the name of the App to something more appropriate. In my case I will change the name of the App to “App: Example Day 3″.

<meta name="Name" content="App: Example Day 3"/>

The final code step is to create and show the Table in the browser.

var table = new rally.sdk.ui.Table(config, rallyDataSource);
table.display("myDiv");

After you add that last piece of code you should be able to open LastDay.template.html in your browser. If you are currently logged into Rally you should see a list of stories that have been updated in a table.

To combine these files into one file so that it can be deployed into a Rally App tab all you need to do is use the starter kit’s combine task.

rake combine

That will create a file named LastDayApp.html, the contents of this file can be copied into a Rally App tab or Custom Tab.

LastDay zip archive is attached to help illustrate the things I covered in the post.

In my next App Tutorial I will show you how to add the ability to show Kanban cards that are colored by the tags they contain.

We use JSDuck to document our JavaScript.  This is the same tool that Ext uses for its documentation.

A few days ago, I was looking through some of our JavaScript where we’d put comments that would be picked up by JSDuck when it creates the documentation that we use.

We had some config options that were incorrectly documented like this:

/**
 * @cfg {String} some string config with a default value
 */
defaultName: 'fooBar'

JSDuck would see ’some’ as the name of the config since it was the first word following the type (String) and ’some’ would show up in the documentation as a config option for the class that was being documented

I first fixed by putting the appropriate name of the config as the first word following the type:

/**
 * @cfg {String} defaultName Some string config with a default value
 */
defaultName: 'fooBar'

Then I noticed that some of the other configs in the same file were defined slightly differently; they didn’t have the name of the config in the comment, but the documentation was still generated correctly:

/**
 * @cfg {String}                                  <-----There's nothing else on this line!
 * Some string config with a default value
 */
defaultName: 'fooBar'

This works great because then you don’t have to copy/paste the name of the config option, JSDuck figures it out for you. Makes it a lot easier when/if the name of the config option gets changed because it won’t be hidden in a comment right above the declaration. Duplication is bad!!!!

At this point, I thought that I’d learned everything I’d ever need to know about JSDuck so I sent out an email to the other developers sharing my new-found knowledge (read: awesomeness )

That’s when the great Mr. Burke Webster added some more great points:

“If your config option has a default value and it’s a basic type, JSDuck will also auto-detect that. So you can do things like this:

/**
 * @cfg
 * Should do something cool
 */
doSomethingCool: true

It will auto-detect that doSomethingCool is a boolean, with a default value of true.

/**
 * @cfg
 * Some string config with a default value
 */
defaultName: 'fooBar'

It will auto-detect that defaultName is a String, with a default value of ‘fooBar’. And if you don’t want to provide a default value, make sure you use ‘undefined’:

/**
 * @cfg
 * A name config for some component
 */
name: undefined

If you try other empty (or falsy) values, like empty string or null, the docs will report the config option has a default value of null.”

Engineering blog: How long have you been with Rally and what were you doing before?
Matt: I’ve been with Rally for about a year and a half now. Previously, I was working as a software developer/architect for the University of Phoenix.

EB: What made you choose Rally?
Matt: I chose Rally because of its: Passion for quality in all levels of the organization. Commitment to community and environment. Focus on employees maintaining a healthy work-life balance. Fun! – This company seriously knows how to have fun.

EB: What sorts of things have you worked on at Rally?
Matt: Converting from Eclipselink to JPA API (btw, thanks for that week 1 hazing!). Migrating Rally search to use the Solr search engine. Foundation of our new User Interface – ExtJS4-based JavaScript toolkit, Multi-Tab browsing. Upgrading our Ruby Selenium browser tests to use the version 2.0 WebDriver API.

EB: What are your favorite things to work on?
Matt: I prefer working on the front-end, enhancing our user experience and developing the building blocks to enable fast development for both our internal teams and external customers building new UI’s with our Web Services API and JS toolkit.

EB: Explain your nickname.
Matt: P-bomb. Originally coined by Russ while playing foosball against me. It typically refers to me scoring on a thundering shot from the defensive position or a slow change-up shot that drives the goalie crazy when they can’t stop it. At that time, p-bombs were seldom seen, but I’ve improved enough that I think others secretly gather to share how terrified they are of them now.

EB: Anything else you’d like to share about life at Rally?
Matt: Can’t really say good enough good things. It’s a great group of passionate people who challenge me and bring out the best in each other.

In my first post I explained the advantages of creating your own App in Rally. An App is a customized view of Rally data that is served in an IFrame. Apps can create interesting ways to interact with Rally data and internally we use them to prototype new features and ideas that are not quite ready to be a portion of the full product. In this post I will explain to you some of the tools that can be used to help you get started on your first App.

One of the challenges facing new App developers is how to organize their files. It is usually easier to develop an App by breaking out the different language sections into their own files. This is because Editors and IDEs tend to be more efficient when working on a file that contains purely HTML, CSS or JavaScript rather than a mixture. Unfortunately Rally expects Apps to be in one contiguous blob of text when it is pasted into the custom App tab. In my experience it can be error prone trying to copy the contents of each style and script tag from an outside file into a single page HTML file.

One of the ways that we circumvent this problem while we develop Apps internally is through the App Development Starter Kit. The starter kit can be found alongside our other tools in our developer portal. It combines each of the separate CSS and JavaScript files together into the one centralized HTML file. This file can be easily copied into Rally to create a new custom App. Once you have the starter kit, making a new App is a breeze. All you need to do is run rake new[APPNAME] and all the files that you need to start your new App will be combined into that contiguous blob the Rally tool knows and loves.

In part two I will show you how to create an App that shows you all of the work items that have been changed in the past day.

I have a love/hate relationship with ORM. I love how it can be a massive productivity boost for some projects. I hate how it’s so easy to misuse. I still firmly believe that if ORM is used appropriately, it has a place in many applications.

Using a mapped collection on an ORM-managed entity is almost never “appropriate use”.

Consider this code. Assume it’s on an entity named Category. You have another entity type named Document. You can add Documents to Categories.

public void addDocument(Document document) {
  getDocuments().add(document);
}

DON’T DO THIS! Ever. Period. End of story. It will PWN your server one day. Okay, really, don’t do it. When the category accumulates a large number of documents, you’ll spend more time allocating memory and querying than is really appropriate. If you’re not sure what I’m talking about, take a look at how mapped collections work in your ORM of choice. They are mostly the same across most ORMs.

If you need to manage a collection-like relationship when using an ORM, always have the child record manage it. If you absolutely need the addDocument method on category, implement it like this:

public void addDocument(Document document) {
  document.addCategory(this);
}

This saves you from having the ORM hydrate the collection when loading the documents collection on category.

There’s one problem with both code samples. If you’re acting on the document or the category in memory, and you don’t manage the collection from both directions, you could end up with inconsistent results. Even worse, if you’re using a second level cache, you could end up with semi-persistent inconsistent results.

Pay attention to how you use collections with an ORM. If you use them, you’re probably wrong.

I’ve blogged about it before. Bang is evil, and by bang I mean negative conditionals. Even when you try hard to make them easy to read, boolean expressions can be error prone.

Today’s installment comes to us thanks to liquibase preconditions. My team ran into a problem yesterday with boolean logic in liquibase preconditions that was specifically related to negation. The migration in question looked something like this:

<preConditions onFail="MARK_RAN">
  <sqlCheck expectedResult="0">SELECT COUNT(*) FROM table_x WHERE column_a = 'XYZ';</sqlCheck>
</preConditions>
<sql>
  UPDATE table_x SET column_a = 'ABC' WHERE column_a = 'XYZ';
</sql>

Can you spot the bug?

The bug is using ‘XYZ’ in the sqlCheck rather than ‘ABC’. We want to change ‘XYZ’ to ‘ABC’ in table_x. Therefore, our sqlCheck is the inverse of what we want. Our migration can never run.

Why did this happen? It happened because the logic is to run a check, then compare the result to zero, then invert the condition and only take an alternative action if we don’t get zero. Mentally parsing the combination of expressions while reading the migration is error prone. It’s easy to mentally negate the expression an incorrect number of times.

The bottom line is that negative conditionals impede the readability of any language. Period.

Engineering blog: How long have you been with Rally and what were you doing before?
Steve Neely: I joined Rally in Feb 2010. Before that I lived in Dublin, Ireland and worked at University College Dublin in the Systems Research Group. Before Dublin I lived in Glasgow, Scotland and worked as a lecturer at the University of Strathclyde.

EB: What made you choose Rally?
SN: I wanted to work with industry leaders in an environment where I could learn and have fun.

EB: What sorts of things have you worked on at Rally?
SN: Across the stack. I started on the Chuck Norris team building the frontend JS that comprises the users page; then was part of Das Schnitzel building Solr search; then as a member of YUNO? I upgraded our Ruby GUI testing infrastructure; and now I’m on the architecture/scalability/performance team (team names: Steel Panther and !pants).

EB: What are your favorite things to work on?
SN: The architecture work we’re doing now is complex and super interesting. A great thing about working at Rally is I’m free to move to any team. I could jump back into JS-land tomorrow if I wanted.

EB: Explain your nickname.
SN: According to one of the guys in the dev team, Groundskeeper Willie, from the Simpsons, is the most famous other Scottish person ever. In my first week they changed my Rally display name and photo. It stuck.

EB: Anything else you’d like to share about life at Rally?
SN: I learn something new every day and love coming to work. That’s an awesome thing to be able to say.