In his book, The Lean Startup, Eric Ries talks about the Build-Measure-Learn feedback loop as the primary way of executing in environments of uncertainty. The goal of this feedback loop is to turn assumptions, risks, and unknowns into knowledge. Knowledge guides teams and companies to progress effectively through uncertain environments, and this leads to a more stable path to innovation. For this reason, the key measure of success is the cycle time of learning -- how quickly a hunch is validated or invalidated.
However, there is a dilemma. Most people who have run the Build-Measure-Learn feedback loop will admit to times when they get to the end of the experiment, scratch their heads, and think that they didn’t learn anything. They had an unexpected result. But that’s actually good news. Learning occurred and the experiment created knowledge. What you don’t want is to get an inconclusive result where it’s unclear what actually happened. This may seem like it’s no big deal, because most people intuitively associate their value with building something. But it is a big deal. In uncertain environments, turning key unknowns into knowledge through this Build-Measure-Learn approach is the primary value-generating activity. You don’t get credit for just building something; you have to have created knowledge.
What To Do
At this point you may be thinking, “Ok, how does my organization up its probability of creating knowledge through experiments?” I think the Build-Measure-Learn loop is missing a critical step: Frame. Running Frame-Build-Measure-Learn loops increases the probability of generating knowledge from your experiments.
There are two aspects to this new Frame step: framing the problem and framing the experiment. Both of these steps are designed to make the rest of the process, Build-Measure-Learn, more effective.
Frame the Problem
Before you start building and running an experiment, you must first set context for your work. That means setting informed boundaries around what your experiment is meant to learn or solve. Start by building empathy for your users and customers. The better your intuition about your customers, the better you will construct experiments to test your understanding.
One of many techniques is Journey Mapping from Design Thinking. This is a great way to holistically visualize your users’ experience. If this process is too laborious, even simple empathy maps can be very helpful. Your organization may have guidelines for reaching out to customers. You might also need to get and set up permission to do this work. Getting sufficient permission and gathering empathy are all part of how you define your problem.
Frame the Experiment
You have a good sense of the problem space; now, it’s time to detail how you will run your experiment.
- Background: what has led you to this experiment? What do you want to learn and why? What information feeds into your work from the Frame step? Write this down; it forces you to articulate what you hope to learn.
- Hypothesis: simply express your expectations for your experiment. This hypothesis is what you repeatedly expect as a result. Without a well-articulated hypothesis, you set yourself up for getting inconclusive results.
- Variables: consider the variables in your environment. Remember your childhood science fair? You identified dependent, independent, and control variables. Make sure there is a clear correlation between the dependent and independent variables.
- Details: document the details of how you will run your experiment. This is important! Use the structure in an experiment sheet. This discipline ensures you’re running an experiment designed for learning.
These steps in Frame are important. Our brains can play tricks on us. One particular trick, retrospective coherence, makes us believe the results we experience were what we expected all along. This occurs even if the results differ dramatically from what we originally expected. To avoid retrospective coherence, write down your expectations before running your experiment. At the end of the experiment, review these. Put yourself in a position to see the unexpected. Seek to learn where invalidation has occurred. These are the moments when learning creates knowledge.
Creating effective learning means you build with discipline and rigor in how you experiment. Pay attention to these three activities in the Build part of your work.
Designing the Experiment
Now that you’ve framed your experiment, you can design, build, and run it. Use an Agile development methodology to build out the details of your experiment. It’s important to consider how everything will fit together. Pay special attention to designing and building the right data collection methods. When the experiment is complete, your data must be robust enough to provide results and actionable information.
Building the Experiment
When it’s time to build your experiment, avoid the pull to create something big and complicated. Remember, you want to build the smallest possible increment sufficient to validate or invalidate your hypothesis with data. Perhaps all you need is a paper prototype or a landing page to collect the data you need.
Running the Experiment
Once designed and built, it’s time to run your experiment and collect your data. This could be as simple as conducting an invalidation interview with your potential customers or as elaborate as shipping software. The key is to work in increments that maximize the number of learning cycles you can run.
You ran your experiment and collected data. Now it’s time to measure the results. When you designed and built your experiment, you included dependent, independent, and control variables that were important to evaluating your hypothesis. Start by analyzing the data that the experiment provided.
What actually happened during the experiment? How do the data relate to your hypothesis? Do the results support the hypothesis or invalidate it? Consider what you expected to happen, as documented, versus what actually happened.
Finally, organize your data and present it as a compelling story. You want to teach and enable your team from the results of your experiment. Simple, engaging stories are the best way to broaden the value of the learning. So, distill your data into a narrative with just the parts that matter. Ensure that your narrative invites listeners to move to creating knowledge via the next step: learning.
Document and Share Your Findings
The final step in the Frame-Build-Measure-Learn loop sets you up to create new experiments or new actions from the measurements you’ve accumulated. Compare your hypothesis and expected results with actual results. What happened? You invested time and money to buy knowledge. What is that knowledge? How should it be preserved? What should your next steps be?
First, complete the Experiment Sheet and share it. Fill it out, including next steps, the cost of obtaining results, and ancillary insights. Sometimes, ancillary knowledge or insight turns out to be the most valuable result of running your experiment.
Make sure that you share your findings in a very transparent way. If you are part of a co-located team, post materials on a wall in your workspace. Otherwise, put your materials online in a shared space where others can see them. This way, your team can easily access the current business model, assumptions, and experiments.
Add Frame. Add Learning.
I believe effective experimentation is key to navigating the uncertainties we face as companies and as a society. If you want to ensure you learn from experiments, Frame before you Build and Measure. As the level of uncertainty continues to increase, our futures depend on us getting good at this.
To learn how “Frame Build Measure Learn” fits into your overall path to discovering your future, visit: www.rallydev.com/frame.