Engineering Blog

I bet you’ve never once done this before right?

Or how about this in prod when you thought it was dev?

Mistakes happen, but they still suck. As time goes on you become pretty careful about how you do things in production environments. Most of us who have been burned have our own tricks to avoid this. Usually this comes down to visual indicators that tell you what environment you are working in. It never hurts to add in a little pause to think before hitting enter as well.

It’s a really simple idea, but one I don’t often see implemented except by folks who have been burned. It’s better if you actually implement it across the organization to help everybody out. Here are a few examples of things we use here at Rally and that I’ve used in the past to help make important information more obvious.

Shell prompt to make the environment obvious

If you login to a production system at Rally – no matter who you are – you’ll see something like this:

And if you su to root you’ll see this:

The Red prompt tells you:

• You are on production, be careful
• The name of the server you are logged into
• If you su to root, we try to make that more evident

This is accomplished by creating a file called /etc/profile.d/prompt.sh with the following contents:

# Special prompt for production systems
if [ "$EUID" = "0" ]; then
  PS1='\[\033[01;31m\]| PROD - \H - ROOT | \[\033[00m\]\n[\w]# '
else
  PS1='\[\033[01;31m\]| PROD - \H - \u | \[\033[00m\]\n[\w]$ '
fi

Similarly – if you login to a non-production demo system you might see something like this:

The yellow prompt tells you two things:

• You are not on production
• You are still on a demo system which might be in use right now, so still be careful.

If you login and see something like this:

You know something is amiss, because Adam shouldn’t have production access.

Leverage your MOTD

A feature of Unix I rarely see used to it’s full extent is the motd. The contents of this file are dumped to the terminal each time you login and are an excellent place for messages you want users to see when they login. Look at the prompt below and then we can discuss:

This message tells us a number of things immediately upon logging in:

• Someone else is doing something with this system
• What date they started work
• Who that person is
• What internal story (S123456) is associated with this work

This is pretty much all you need. You can look at the story if you want more detail, you can follow up with the individual to see what’s going on, but most importantly, if an alarm went off in the monitoring system for this system you know it’s not in service and you aren’t going to attempt to “fix” it without checking with the person listed.

I like to refer to this as “communicating close to the change”. Try to add an indicator somewhere that others will see it during the natural course of their work looking into a change. Having this information in a change log somewhere is great but if an alarm goes off at 3am, I’m just going to login to the system and take a look at it – having the motd in place assures I’ll see it.

Change the color of your screen/tmux status bar

I tend to use screen or tmux when I’m working on systems. This gives me one additional area where I can expose an indicator of my environment.

So now, not only do I have my shell prompt to tell me I’m in production but I have the red bar at the bottom telling me I am in production. For contrast, here is what my tmux status bar looks like when I’m not in production:

You can change this pretty easy in tmux.conf:

Here is production:

# Highlighting the active window in status bar
setw -g window-status-current-bg red
setw -g window-status-current-fg white

And here is non-production:

# Highlighting the active window in status bar
setw -g window-status-current-bg green
setw -g window-status-current-fg white

Wrapping up

The main point here is that you should make it as obvious as you can (without being intrusive) which environment you are working in. Use colors, blinky lights or whatever else you need to make sure you know where you are working at all times. This doesn’t prevent all mistakes, but it makes it easier to avoid some.