Engineering Blog

One of the (many) great things about Rally culture is the Hackathon concept. We get a period of time (traditionally it was one week in every 8, this time out will be longer due to a strategic rescheduling) to work on a project of our choice, with the caveat that the project will benefit the company. Pretty wide remit, huh? We love Hackathon as it gives us time to work on pet peeves and/or pet projects we might not otherwise be able to. It’s a chance to work with people outside of your normal team, learn a new language, build something really cool, prototype designs in a safe environment, etc. In short, it’s awesomeness.

Our current performance tests are written in Java and run via JMeter but they don’t actually give us true client side metrics since they don’t spin up a browser. We know how well the Java performs, but (as an example) all that JavaScript we’ve been writing about may affect our application performance for whatever reason. In short, we won’t really know for sure how fast the system is or isn’t until after the code is released and production monitoring takes over. Unsurprisingly we aren’t particularly comfortable with that, and want to get a holistic view of our overall system performance prior to release. One way we think we can get that view is by spinning up browsers and exercising the application as a user would. Since our browser test framework already spins up 100s of mechanical ‘users’ to exercise the application, being able to leverage it to track ‘real world’ usage metrics should give us a valuable insight into how the system really behaves.

We’ve invested heavily in the framework to make it better, and during that investment we spent time refactoring it in a way that makes it fairly easy to re-purpose certain aspects. That’s a story for another post, but put simply our tests use generic behaviors that call the implementation specific driver for the page or panel. Last time out The Chauncey and I started some work re-purposing our existing browser test framework to be used in performance tests. We had a good proof of concept: JMeter was calling the Ruby scripts, parameters were passed between the right places, and some of the appropriate metrics were being written to the appropriate logs. Then we ran out of time and had to go back to work… But we didn’t throw away what we had, instead we tucked it away in a branch for safe-keeping with some notes about what was outstanding, and what needed some more work.

In case you haven’t guessed by now, our plan is to finish up that work in this coming Hackathon. We plan to take the drivers and write a collection of performance behaviors to give us more bang for the bucks we spent on the tests and gain that holistic view. Furthermore, our current performance tests have a fairly large feedback loop, so I’m also wanting an improved execution time so we aren’t suffering from a lack of attention to performance simply due to the time it takes to get results.

There are some hurdles to overcome however, our framework is very much geared to functional testing in the Arrange/Act/Assert pattern but to gather “real world performance” numbers we’ll need to write tests that can use pre-existing data. Currently our tests are data-independent in that each one creates _everything_ it needs via the WSAPI endpoint. In order to get meaningful metrics from the tests they’ll need to use large datasets, so we’ll have to make it possible to pass a user to a test and then manipulate the data available to that user, and then do the same operations for a different user and it’s respective data. So I figure it’ll end up being more Collect/Act/Assert (I’m fairly sure I made that up but hopefully you get the idea). Perhaps the assert portion will simply be along the lines of !exception, I’m not quite sure how it will look yet. We might even end up with functional performance tests, where the two are combined and we measure our functional tests – while that concept doesn’t feel quite right to me while I’m writing this, I’m open to the possibility.

Regardless, I’m pretty excited about the chance to crack on with this project. I’ll report back once the Hackathon is over, or if the project [fails | reaches it's minimum capability state and we can think about plans for putting it into production].