Engineering Blog

In my last post, I gave a brief rundown of why we use feature toggles. At the end of that post, I made a list of questions people often have about feature toggles. Ross made a good point in the comments that hinted at the answer to one of my questions. Because I haven’t received any other suggestions, and because his comment pertains to my favorite part of dealing with feature toggles, I’m going to write more about that.

At Rally, the usage of feature toggles varies according to the type of work that is being toggled. Because my team focuses primarily on infrastructure work, we most often use feature toggles to branch between two pieces of code that do the exact same thing from the outside world’s perspective. Typically, we are doing this because the old code does something undesirable. By the time the code has been in production and has been proven stable (or stable enough), my team is usually itching to get busy deleting code.

We handle the removal of toggles and the associated code as a follow-up user story to the original. Before we even release the initial code with the toggle into production, we create the remove toggle story near the top of our ready to pull queue and mark it as blocked. By keeping it near the top of the queue, we are constantly reminded that it is there. By blocking it, we are signaling that the work cannot procede. The combination of the two forces us to focus on the toggle story, which is further to the right on our board.

This is a bit of a non-sequitur, but I want to focus on that last sentence. Specifically, I want to focus on what it means to focus on the right side of the board. Our team uses Kanban and we track our work in Rally using the Kanban Board app. This is a dramatic over-simplification, but Kanban encourages you to reduce the amount of work that is in progress and focus instead on pulling across the work that’s already in progress. Cards move from left to right across our board, and at any given time the greatest value is moving whatever is furthest to the right even further (or off the board entirely).

Our board has columns specifically focused on toggled work. Work items can move into “Ready to Toggle On” when all code for the work item is in production and the work has been accepted. Work items then move into the “Monitoring” column as soon as we can get our ops team to toggle it on for a subset of users. We try to do this as soon an open slot appears in “Monitoring”. Once the work has been toggled on for all users, it gets moved into “Release” which is the same column that non-toggled code ends up in once it’s pushed to production.

We intentionally keep the limits on the number of items we can have in those columns small to prevent having an explosion of partially toggled features. The WIP (work in progess) limits for each column can be seen in this screenshot:

Because our git configuration makes it pretty obvious how many lines were added or removed in a commit, the actual work of removing toggles becomes a game. Chauncey hinted at this in his comment on my original post, but for every story we do something called a “chinese auction” to set the line for a friendly wager. Usually the loser buys the winner a beer.

Because the wikipedia definition of a chinese auction doesn’t match what we do, let me explain. A chinese auction is a specialized form of an over/under bet where you can only take the under. The two developers who are about to make the commit take turns choosing how many lines they think will be deleted until the other developer “takes the under”. I might guess 100 lines will be deleted, and if Chauncey thinks it’s less than that he can take the under or suggest a new (larger) number, at which point I’ll have the same choice.

In the most interesting such wager I can remember, I chose 2500 lines for a story after a bit of back and forth with Eric. He took the under and won the beer when the commit came in at 2496 lines deleted.

Feature toggles are an awesome resource, but it’s important that the toggles and the old code be removed as soon as possible. By making the remove toggle story a game, we keep our motivation up. By making toggles an explicit part of our Kanban board and the associated policies, we stay focused on the need to remove the toggles.