Engineering Blog

We have a new component that went into production 2 weeks ago. It had been under development for about 2 weeks when we got it running on production servers. At this point, the application itself does one thing – says “Hello World”. What had we been doing those two weeks before we first deployed it?

• Evaluating & selecting the technology for the new component.
• Determining packaging of the application
• Building startup scripts
• Building deployment scripts
• Testing the deployment in test environments.

This application deployed to production two weeks ago onto new systems and will not see any production traffic for a while – it still only says “Hello World”. It will continue to live on VM hosts until we determine that it has specific hardware requirements.

Why is this being done? Because there is a lot we do not know and the sooner this application is in production the sooner we can get answers to those questions. We have performance testing environments that work pretty well for us – but there are lots of peripheral issues to work out when you get a system into production and we wanted to work through those early.

We have a continuous build environment and fully automated weekly deploys to production. We use feature flags and branch in code to control code availability in production. The hard part of building a new component is introducing it into that weekly flow without causing any problems – so in this case we are solving that problem first. Now that the application is in production it should be minor tweaks to build it up to the point where it’s getting production traffic. Each incremental change that’s made will go into production within the week – if Developers want to start diverting some traffic to this component for testing, they can do that with very little interaction from Ops (feature toggle most likely). Operations can begin to monitor the application, deal with logs, start collecting performance metrics – all before this application sees customer traffic.

When we get to the point where the application is fully feature complete Ops should have a pretty good handle on knowing if it’s working & have worked out any kinks related to it’s production readiness.

The caveat to all of this is that you must have mechanisms in place to make sure this new system isn’t going to cause harm in your production environment. This application goes through the same functional & performance testing environment every day that all of our applications do. From the very beginning, even though it only said “Hello World”, there was a performance test to exercise the query that produced “Hello World”. For every incremental change we make to the application, tests will be built to make sure that new functionality works as expected before it hits production. As new features are added to the application, things like DB connectivity, Ops is notified & we’re monitoring the impact these new changes have on the overall system. Because the change is incremental it’s a lot easier to identify any problems and respond to them.

There are a lot of benefits to this approach for new code:

• Ops gains comfort with the new component very early & can provide feedback to Developers while it’s being built when there is the least possible resistance to change.
• Rather than guessing at hardware requirements you acknowledge that you aren’t sure and deploy in the least costly way and get feedback on performance as early as possible.
• Developers can test in production very early to catch issues that simply are not exposed in our testing environments.
• Operations has a lot more time to get configs into our CMS (Puppet), get monitoring setup, and get log ingestion setup before customer traffic is flowing through the system.
• As the product is developed, Operations is giving status at stand-up about the production environment readiness & any problems encountered.
• Everyone is happier because they are working together from the beginning to make this happen.

Not every component we build goes out this way – and in fact, it’s an uncommon event for us to add components. So, how did we end up getting things prioritized so that the deployment tasks were done up front?

• Everyone agreed that getting the application into production successfully was a critical part of the development process – and sorting it out early had a lot of benefits.
• There were a lot of questions about what the software would need to do & how it would perform and our answers to those questions needed to be validated in both testing & production environments before the system went live.
• The Dev team had design & testing to do before they could start building a system – this time was used for the build/release/ops teams to get moving on the deployment configurations, packaging & target systems.
• An Operations team member was present from the very first stand up for this team.
• The build/release team was present from the very first stand up for this team

This may not work for you, but we think it worked pretty well for us. As the system is developed and becomes an integral part of our system receiving production traffic, we can let you know how this whole process contributed to the quality of the product. For now, it delivers a damn fine “Hello World”.