Engineering Blog

Mark Levison recently posted a challenge to Agile tool vendors, so I’ve written some responses below about our Definition of Done, TDD, acceptance testing, release frequency, and retrospectives.

Definition of Done
We separate out our definition of “Done” for individual stories and for the release as a whole.  Each release planning, we discuss it as a group and make updates, so this is current as of last Friday:

Story
All tasks complete
All tests running and passing
Manual walkthroughs complete on all fully supported browsers
Migrations and Web Services updates reviewed
Ops & On-Premise impact recorded
User gesture tracking added
Performance test results reviewed
All defects closed by testers
UX review complete
Accepted by Product Owners

Release
All stories accepted
All tests run & passing
Build passing
Performance tests run & accepted
Migration tested against trial & production
Outage time defined & communicated
Doc & external content complete
Agile Commons (our user community)
Announcements (pre & post)
Release notes
“What’s New?” screen updated
Interstitial light box

TDD and Unit Testing
We do TDD to different extents in different parts of the system.  TDD in Java has been relatively straightforward, but as a rich web application, we have fairly sophisticated JavaScript code that historically has been tricky to unit test.  In the past, we leaned heavily on our acceptance testing framework on the GUI side, but we’ve recently come up with a JavaScript unit testing approach that we’re happy with and that supports TDD.

Acceptance Testing
For the last few years, we’ve been writing tests in an internally built Ruby framework that drives Selenium for much of our automated functional acceptance testing.  This works, but the tests take a long time to run, and so we’re exploring other strategies.

We do support a lot of browsers for a relatively sophisticated client-side experience.  Firefox 3.x, IE 8.x, Safari 4.x, and Chrome 2.x are all fully supported.  For older browsers (Firefox 2.x, Safari 3.x, IE 7.x and 6.x) we provide functional support – Rally needs to work on these, but sometimes there are display issues with margins and the like.  We have one exploratory tester per scrum who does exploratory testing on each story on each of these different browsers.

We also do automated performance testing on each story to see the impact on overall performance, to be certain that new work doesn’t degrade our response times.  Because many of our customers are international, we include user gesture tracking so we can keep track of actual client-side performance on different browsers in different parts of the world so we know what performance users are experiencing after we release.  Our average server-side response time is under 200ms, and our average client-side response time (server time + network time + browser render time) is under 1 second for the top browsers.    Here’s some actual data for real customers this fall:

Finally, the product owners also do a manual walkthrough of each story as it is completed.  The team has already verified that the acceptance criteria are tested, but sometimes during the acceptance process I find missing acceptance criteria that turn into additional stories.

Release Frequency
We release about 40 times per year.  Patches with small features and fixes go out most Saturdays, and we have major feature releases 6 times per year so we can manage customer expectations and keep the rest of the company in sync.

Last Retrospective
I feel like revealing our retrospective contents in public might violate trust and a feeling of personal safety for our team.  One issue that came up recently was our test system and the extent to which it mirrors our production environment.  Each scrum holds a retrospective every other week, and we do a cross-scrum retrospective every 8 weeks.

But really, these things are table stakes – you’re probably going to get decent answers from most agile tool vendors.  There are other questions that seem to matter a lot more to our customers when they’re choosing a tool

- How well does the vendor match the level of engagement that you want?  If you want a deep, “partner” level of engagement, can you get it?  Or is the vendor going to dump a zip file in your lap and ignore your emails?
- How smooth are your releases?  Will we have to spend a lot of time adjusting how we work each time you change things?  Or do you put effort into building features that are easy for users to adopt and non-disruptive?
- How’s your performance internationally?  If we have teams all over the world, what actual client-side performance do users of your product experience in Mumbai?  In Singapore?  In Chicago?