Engineering Blog

Ever feel like your Ops team just doesn’t understand what’s going on in Development? Ever feel like your Development team is just “tossing stuff over the wall”? I’ve felt like that plenty in the past. There are all kinds of strategies to battle this, this post will talk about some of the things we do at Rally to keep things moving smoothly.

Some of these are dedicated meetings and some are just Ops being involved in the regular Development process. In general the idea is to try to have Operations and Engineering in the same room as much as possible when decisions are being made.

Stand Up Integration – Daily, 5-15 minutes

We have many different Dev teams working on a variety projects at the same time. That makes for a big challenge staying in touch with all those projects in Ops. One of the most effective ways that we have found to get visibility into what is going on in Dev is to attend their daily standups. This doesn’t require a ton of our time, usually 5-15 minutes per team, and has a huge benefit.

We try to have one person in Ops assigned to a particular stand-up and work with Development to stagger their stand-ups so that we can attend them. Ideally the same person in Ops goes to the stand-up each day.

A few things attending standup each day buys us:

• Get to hear about new changes as they are being concieved & built.
• Ability to gather feedback from Ops & Dev each day and keep both teams in sync as ideas evolve.
• As issues come up in production for a given team they can be brought to the Dev team directly by someone from Ops who is familiar with their projects because they attend the stand-up each day.
• Establish a relationship where the designated Ops person works as part of the Dev team. They aren’t necessarily responsible for all the tasks that Dev team needs, but they are an easy and available point of contact.

This doesn’t solve 100% of your problems, but it does help to open up communication & involvement between the teams and goes a long way toward getting things moving smoothly.

SoS (Scrum of Scrums) – Daily, 5-10 minutes

This typically happens following most of the individual team stand-ups. This is a very quick meeting for teams to raise any cross-team dependency issues. If Ops had a production issue the night before, we let everyone know at SoS. If there are problems that are blocking another team and they need help, they get raised at SoS. If there’s an important demo going on and we need to be extra vigilant, it gets raised at SoS.

SoS is a fantastic opportunity to make everyone aware of important current events. If there are issues that are raised which require additional discussion we typically have folks stay behind who want to discuss and that meeting happens immediately following SoS.

Dev/Ops Sync – Weekly, 1 hour

Once a week we pull the Operations & Engineering teams together for a meeting we call “Dev/Ops Sync”. This meeting has evolved over time but the basic premise & agenda is something like this:

• Discuss changes in this weeks release, what features are being toggled on, what does Ops need to be aware of with the upcoming release.
• Talk about any open issues we have for the Dev team. This could be defects we have created which are impacting production, new issues that have been discovered since the last meeting, or areas of concern where we want input from Developers to help steer our actions.
• Make Development aware of any outages & share the associated PER document.
• Talk about any issues Development needs to bring to Ops. Some of these issues are headed off by Ops attending stand-ups but sometimes there are larger issues that need to be discussed.

We try to have someone from each of the Engineering teams represented here which helps facilitate making sure we can cover any questions that come up for any of the teams.

Architecture Council – Weekly, 1 hour

This meeting isn’t strictly for keeping Dev and Ops in sync and more often than not it’s pretty Dev focused, but Operations involvement in this meeting is always significant. The Architecture Council at Rally is where our larger architectural issues are raised, discussed, and assigned to teams for prioritizing the work. This team is intended to make sure there is a forum for raising issues that could cause us problems down the road, for reviewing the design of new features and components being built, and to provide a way to keep all teams in sync around what important issues are being worked on.

Sometimes this team meeting is where decisions get made, sometimes it’s just to review what decisions are being made, but it’s one of the areas Dev & Ops come together to get consensus on how to move forward.

Retro & Demo – Bi-Weekly, 1 hour

Again, not strictly for the benefit of Ops, but these are two meetings where Dev and Ops come together to talk about how things are going. Every week on Friday we have either an Engineering Demo or a Retrospective. Both of these are opportunities to share what work is being done and what’s working or what isn’t.

The Engineering Demo

This is where our Development Team shows off what they’ve been working on. Sometimes it’s Will showing us some awesome new menu’s, other times it’s Jeff showing us an app that says “Hello World”. Either way, Dev & Ops get to ask questions and give feedback on work in progress, hang out and relax, and have fun in the process.

The Retrospective

Our Bi-Weekly retro’s are where each team reflects on what worked and what didn’t over the previous 2 week period and share those items that they think are important for the other teams. This is a great opportunity for Ops to raise examples of things that worked well and to call out areas where we need help. It is also a chance for any of the teams to raise items that aren’t working well and get those items assigned to an owner so they get fixed. It’s not perfect, some things take time to address, but if something continues to be a problem it continues to be raised here and usually gets fixed sooner than later.

Do what works for you

These are the things that are working for Rally today, but we change all the time and try new things. You may not need all of these meetings to have a successful Dev/Ops culture but hopefully some of these ideas help. The most important part of all of this is to foster communication and involvement by both teams – making team members accessible & constantly asking “what can we do better?”.