Co-creation and research collaboration with the Software Engineering Institute and Rally’s enterprise-class customers

Today Rally announced the formation of the Software Development Performance Index research program. Representatives from Rally’s enterprise-class customers, the Software Engineering Institute (SEI) at Carnegie Mellon University, and Rally are joining forces to quantify valuable insights about development work and to provide feedback on how process and technology decisions impact development team performance.

In the popular film The Matrix, Neo is given the choice between the “blue pill” representing the blissful ignorance of illusion, and the “red pill” embracing a sometimes-painful reality. The choice between a well-informed decision or a poorly-informed decision can make all the difference between success and failure for a product or a company.

Take the Red Pill (photo by CC license from Roman Soto)

What Rally is launching this week will attempt to formulate the “red pill” for software development: to identify the tools and techniques that will provide quantitative insight upon which to base business decisions.face

A Perfect Storm of Interest, Opportunity, and Collaboration
Things have come together in a perfect storm which has allowed us to formally launch this program. A couple of years ago, I left my full-time Ph. D. pursuits at Carnegie Mellon University (CMU) to get closer to real-world development with the intention of finishing my Ph. D. part-time. However, my role as the Product Owner of Rally’s analytics team did not allow much time for research!

This past January, I received a fellowship at CMU from Los Alamos National Labs, which allowed me to return to the research part-time. Around the same time, it became clear to me that the software development industry was sorely lacking this data and that (due to our multi-tenant platform) Rally was uniquely positioned to provide it.

So in June, we started working towards the launch of this research initiative at Rally. I reached out to my friends at the SEI (located at CMU) because I knew they were engaging in similar research and we recently signed an agreement with them to assist us in this research.

My position as Product Owner at Rally was backfilled last month by the very capable Allen Barksdale. Allen previously worked with the core analytics developers prior to the Sixth Sense acquisition and we were glad he was available to help us out again. I will still be involved in the leadership of Rally’s analytics vision as Rally’s Director of Analytics, but I am now free to build a small team to conduct this research and manage this co-creation program.

The new team is appropriately named “Red Pill”.

Goals and Deliverables
What are we doing and why are we doing it? Our intention is to:

1. Validate a balanced set of outcome measures that can be easily extracted from sources such as application lifecycle management tools, source code repositories, collaboration tools, and lightweight surveys.

2. Identify the scope of comparability for various measures to help you decide which measures to use for team trending and which might be appropriate for cross-team or cross-organization comparison.

3. Show the correlation between decisions and these outcome measures, which will identify levers that you can use to impact future results.

4. Provide a framework to implement a healthy metrics regime within your organization.

The first set of deliverables from this effort will include:

1. A research paper demonstrating the efficacy of various agile practices.

2. Prototype products building upon the Performance Index framework in the form of dashboards, balanced scorecards, and benchmarking tools.

Rally is Uniquely Qualified to Lead this Initiative
Due to the nature of Rally’s multi-tenant SaaS platform, Rally is in a unique position to aggregate information from across many of the world’s largest software-driven companies to prove or disprove many commonly held assumptions about software development at scale.

For instance, the Agile community has long advocated for dedicated, co-located teams consisting of between five and nine members. Research results, for example, will quantify to what degree productivity, responsiveness, predictability, and quality are impacted by wholly or partially following these recommendations.

What’s in it for you?
Rally customers will benefit from this research because it will provide insight upon which to base business decisions and guidance on managing the software development process. Rally’s software will be strengthened from the incorporation of these concepts into the product.  The entire Agile community will benefit from the broadly applicable insights and correlations that we discover as a result of this work.

However, the select Rally customers who join the co-creation and research program will accrue additional benefits:

1. The opportunity to have your hypotheses validated. You will be part of a small group of thought-leaders that will assemble the list of hypotheses to investigate.

2. Access to the resulting analysis for your company and context. Maybe the benefit of practice A is a 50% improvement in the general context but for your teams, it’s a 100% improvement? Maybe you want to know how your organization compares to the general context on some outcome indicator? We will provide custom runs of any analysis that we develop to organizations that are part of this collaboration program.

3. Custom development without the custom development cost. The analyses that result from this work will be developed into dashboard panels. The first versions of these will be co-created with program participants and be custom-made from your input, without you paying the cost of custom development.

4. Early access to research and publication opportunities. Neither you nor your company will be identified without your permission. However, if you desire, you will have the opportunity to co-author, co-present at conferences, or just have your name associated with this thought-leadership research.

5. The opportunity to influence Rally’s future product decisions. It is expected that we will add capability to Rally as a result of this work (Benchmarking, Balanced scorecard, Executive dashboards, etc.).

To qualify to be a part of this research study and receive these benefits, you must:

1. Be on our SaaS stack.

2. Be willing to share demographic data (programming language, age of code base, team size, etc.) for a statistically significant portion of your teams.

3. Be willing to allow our researchers see the demographic data and to have it correlated with team metrics extracted from Rally.

We plan to ramp up slowly with three to five customers in the first round and no more than 20 total as the work evolves. We are excited to bring this important research forward and we invite you to be a part of it!

If you meet the above criteria and would like to apply to be a part of this research program with Rally, please contact me at Larry.Maccherone@rallydev.com.

Request a Call

Looking for support? Open a support case.

Send Us Your Feedback

Provide us with some information about yourself and we'll be in touch soon. * Required Field