Engineering Blog

As a user experience professional, my job is to provide development with highly-usable designs that best match the goals of our customers.  But in the fast-paced world of agile software development and being the single UX resource for several teams, I am not able to put every design through the same rigorous User-Centered Design (UCD) process in which I was trained.  Instead, I must analyze the risk of each upcoming feature and determine my approach.  As part of this risk analysis, I consider many of the following questions:

• Is this an additive feature or a change to an existing feature?
• How disruptive will it be to the user’s current process?
• How often will our users use it?
• How many of our users will be impacted by it?
• How critical is the functionality to our users?
• How complex is the design and is there potential for many different alternative designs?
• How well do we understand the goal of the feature and how it will be used?
• Have we already done something similar to this that we can adapt for this purpose?
• Is there another application that has a good design that we can adapt for this purpose?
• How difficult is it to change the design if we get it wrong? (from a user acceptance perspective)
• How difficult is it to change the design if we get it wrong? (from a development effort perspective)

If I determine a feature to be high-risk based on these criteria, I will dedicate a greater amount of my time to bring it through the UCD process.  For a low risk feature, my effort will be greatly reduced.

For example, a few releases ago, we were working on a feature to allow subscription administrators to override user’s individual session timeout settings in order to match their internal security policy.  This feature was primarily additive and the goal very straightforward.  The number of users that would interact with the feature would be a very tiny percentage of our user base.  The frequency that the administrator would change this setting would be low.  The design was simple and well-contained and the possible permutations on the design were relatively few.  Overall, this feature felt low risk.  As a result, I spent about an hour iterating on the design and did not perform any user validation.

In contrast, in the next release, we’ll be working on a very high risk feature – moving our Iteration Status page over to our new page framework.  It’s high risk for several reasons.  First, it’s the number one most used page in Rally and the usage of this page spans across our entire user base.  The possible design alternatives are myriad.  Even small interaction decisions would have a huge impact on user acceptance.  Finally, making significant changes after-the-fact would be costly from a development perspective and would result in significant re-learning from our users.

For this page, my work started about 2 releases out (about 3 months).  I iterated through many designs, conducted internal testing, had interactive prototypes built, tested these prototypes with customers and iterated again.  This work is still underway, with implementation beginning in about 4 weeks.  We will be further mitigating the risk involved by releasing this redesign as a Beta program, allowing us to gather and incorporate feedback before releasing the page to all users.

Many features fall somewhere in between these two extreme examples.  Based on the circumstances, I will draw on different UCD methods and techniques ranging from very lightweight to very heavyweight.  I might rely on feature request comments from our customer forum for one feature but for another feature I’ll schedule one-on-one calls with customers.  Sometimes I’ll perform usability tests with internal users and other times I’ll schedule formal usability tests with external customers.

Using design risk analysis to drive effort allocation is critical to helping me avoid delaying our development team.  With that said, because I come from a traditional UCD background, it can feel “wrong” to not fully validate each design with users.  Fortunately, the “release early, release often” mantra of agile is itself a risk mitigation technique.  In the case that a design is not well-received, we have an opportunity to listen to user feedback and release a better design in just a few short weeks.

As a user experience designer at Rally, I’m thinking about 3 things at any given moment:  1) the work that is being developed right now in the current iteration, 2) the work that’s coming up next iteration and 3) the work that is coming up in the next 1-2 releases.

The Current Iteration

In order to avoid blocking the development team, my first priority is making myself available to answer questions and remove ambiguity from designs and acceptance criteria that may come up once implementation begins.  And for every user story that impacts the UI (most of which do), I have an explicit “UX Review” task.  To complete this task, I sit with the developer and review the work that was done.  For example, I just sat down with one of our developers this morning and walked through an error message dialogue that can appear after a bulk edit is made.  There were a couple of slight alignment and spacing issues that we identified and Katie quickly fixed these.  If the problem is significant and requires a large amount of development effort or rethinking the design, I work with the product owner to create a new user story to capture the additional work.  The new user story is then ranked accordingly – either to be done in the next iteration or sometime later.  Once the UX task is complete, our tester can perform a manual walk-through on all of our supported browsers.

The Next Iteration

My next priority is the the next iteration.  I frequently review the Backlog to identify what we will likely be working on next and as an extra measure I schedule a weekly meeting with the product owners to formally review the set of candidate user stories for the next iteration.  The assumption that the items are listed in rank order is critical to my work and ensures that I am creating detailed designs and acceptance criteria for the right set of work.  It would be wasteful for me to create detailed designs for work that may not be done for several weeks and it would much worse still for me to enter in to an iteration without a solid design for the team.

It’s a delicate dance, however.  Since story size is often dependent on the design and what will fit in an iteration depends on size there is often quite a bit of negotiation that goes on between developers, product owners and myself in preparation for the next iteration.  For example, if I create a design option A that is estimated at 10 points and design option B that is estimated at 5 points, we will weigh the options carefully.  Our developers are very accustomed to me asking them to take a look at a design and tell me how difficult it would be to implement.  Very often, they will suggest alternatives that would be technically easier and if an alternative is equally user-friendly, I’m happy to adapt the design.

On occasion the technical difficulty of a particular design does not come out until the formal process of iteration planning.  In these cases, the end result is usually compromised because the adapted design has not had the opportunity to be iterated on or validated.

The Big Picture

I have found that it’s imperative for me to know what is coming in the next 1 to 2 releases for two reasons:  1) to bring a holistic vision in to each design that I create and 2) to be able to plan ahead for larger, riskier features.  Fortunately, the product owners that I work with hold a Product Council meeting every 2 weeks with stakeholders and this allows me to gain insight in to what we will likely be working on in the next couple of releases.  When I’m not working on the current iteration and the next iteration, I spend my time thinking about these upcoming features and themes.  I identify which features will need customer research and validation and start to schedule these.  I explore high level design concepts and validate them.  This prep work ensures that I won’t be scrambling to get designs together when the work gets scheduled into an iteration and it also helps create a cohesive user experience over the long term.