I bet you’ve never once done this before right?

Screen Shot 2011-11-22 at 11.29.11 PM

Or how about this in prod when you thought it was dev?

Screen Shot 2011-11-22 at 11.31.48 PM

Mistakes happen, but they still suck. As time goes on you become pretty careful about how you do things in production environments. Most of us who have been burned have our own tricks to avoid this. Usually this comes down to visual indicators that tell you what environment you are working in. It never hurts to add in a little pause to think before hitting enter as well.

It’s a really simple idea, but one I don’t often see implemented except by folks who have been burned. It’s better if you actually implement it across the organization to help everybody out. Here are a few examples of things we use here at Rally and that I’ve used in the past to help make important information more obvious.

Shell prompt to make the environment obvious

If you login to a production system at Rally – no matter who you are – you’ll see something like this:

Screen Shot 2011-11-22 at 11.19.18 PM

And if you su to root you’ll see this:

Screen Shot 2011-11-22 at 11.12.13 PM

The Red prompt tells you:

  • You are on production, be careful
  • The name of the server you are logged into
  • If you su to root, we try to make that more evident

This is accomplished by creating a file called /etc/profile.d/prompt.sh with the following contents:

# Special prompt for production systems
if [ "$EUID" = "0" ]; then
  PS1='\[\033[01;31m\]| PROD - \H - ROOT | \[\033[00m\]\n[\w]# '
else
  PS1='\[\033[01;31m\]| PROD - \H - \u | \[\033[00m\]\n[\w]$ '
fi

Similarly – if you login to a non-production demo system you might see something like this:

Screen Shot 2011-11-22 at 11.20.44 PM

The yellow prompt tells you two things:

  • You are not on production
  • You are still on a demo system which might be in use right now, so still be careful.

If you login and see something like this:

Screen Shot 2011-11-23 at 9.48.30 AM

You know something is amiss, because Adam shouldn’t have production access.

Leverage your MOTD

A feature of Unix I rarely see used to it’s full extent is the motd. The contents of this file are dumped to the terminal each time you login and are an excellent place for messages you want users to see when they login. Look at the prompt below and then we can discuss:

Screen Shot 2011-11-26 at 6.56.33 AM

This message tells us a number of things immediately upon logging in:

  • Someone else is doing something with this system
  • What date they started work
  • Who that person is
  • What internal story (S123456) is associated with this work

This is pretty much all you need. You can look at the story if you want more detail, you can follow up with the individual to see what’s going on, but most importantly, if an alarm went off in the monitoring system for this system you know it’s not in service and you aren’t going to attempt to “fix” it without checking with the person listed.

I like to refer to this as “communicating close to the change”. Try to add an indicator somewhere that others will see it during the natural course of their work looking into a change. Having this information in a change log somewhere is great but if an alarm goes off at 3am, I’m just going to login to the system and take a look at it – having the motd in place assures I’ll see it.

Change the color of your screen/tmux status bar

I tend to use screen or tmux when I’m working on systems. This gives me one additional area where I can expose an indicator of my environment.

Screen Shot 2011-11-28 at 10.27.54 AM

So now, not only do I have my shell prompt to tell me I’m in production but I have the red bar at the bottom telling me I am in production. For contrast, here is what my tmux status bar looks like when I’m not in production:

Screen Shot 2011-11-28 at 10.35.04 AM

You can change this pretty easy in tmux.conf:

Here is production:

# Highlighting the active window in status bar
setw -g window-status-current-bg red
setw -g window-status-current-fg white

And here is non-production:

# Highlighting the active window in status bar
setw -g window-status-current-bg green
setw -g window-status-current-fg white

Wrapping up

The main point here is that you should make it as obvious as you can (without being intrusive) which environment you are working in. Use colors, blinky lights or whatever else you need to make sure you know where you are working at all times. This doesn’t prevent all mistakes, but it makes it easier to avoid some.

Request a Call

Looking for support?

Send Us Your Feedback

Provide us with some information about yourself and we'll be in touch soon. * Required Field