Engineering Blog

Recently, at Rally, we made an effort to speed up our tests.    This is an ongoing battle that I suspect all organizations face.    We wanted to minimize the amount of data created by our integration tests.   To minimize the data, we decided to share expensive data across tests.

Sharing data caused several of our integration tests to fail.    Lucky for us, all the problems shared the same symptom; bad assertions.    Take the following example.

 createUnScheduledUserStory();
 createScheduledUserStory();
 createUnScheduledUserStory();
 commit();

 assertCriterionReturns(new UnscheduledCriterion(), 2);

Is this assertion good?    Does it ask the right question?    This assertion failed when we started sharing, but I would like to argue that it was not good in the first place.    Do we really want to know that two items were returned from the query?    No.    We really want to know if the two we created were returned from the query.

public void unscheduledStoriesShouldBeFound()
{
  UserStory[] unscheduled = {
      createUnScheduledUserStory(),
      createUnScheduledUserStory()
  };
  commit();

  assertCriterionFinds(new UnscheduledCriterion(), unscheduled);
}

public void scheduledStoriesShouldNotBeFound()
{
  UserStory scheduled = createScheduledUserStory();
  commit();

  assertCriterionDoesNotFind(new UnscheduledCriterion(), scheduled);
}

This is how the test now exists.    This works when sharing data across tests and it asks a better question.   Now we have no doubt that our criterion returns the expected data.

Lesson learned; Create specific assertions.