Though I don’t follow baseball much, I do know enough that stats are a big part of the game. Thus this anecdote in Michael J. Mauboussin's article "The True Measures of Success" in the Harvard Business Review is pretty interesting:
"Baseball managers used to focus on a basic number—team batting average—when they talked about scoring runs. But after doing a proper statistical analysis, the A’s front office recognized that a player’s ability to get on base was a much better predictor of how many runs he would score."
They go on to recruit players with this ability, and do well by it.
What is interesting about this anecdote is that baseball teams aren't the only ones to potentially miss the mark with their stats. Many activities and sectors seem to follow metrics and models that may not reflect an adequate understanding of how value is created.
Sometimes this can have a much broader social consequences. You only have to consider what financial risk analysts were and weren’t looking at in the lead up to the 2008 financial crisis. According to McKinsey & Company’s Working Paper, “Risk modeling in a new paradigm: developing new insight and foresight on structural risk” (2011), most financial institutions failed to capture structural risk metrics in their management information systems or to quantify their potential impact on daily P&L and balance sheets – the real indicators whether an institution is doing well or going bankrupt. Risk models also overemphasized historical data and failed to detect problems that should have been recognized as advance warning of the looming crisis.
What is the role of measurement? Ideally, measurement creates data for learning and evidence-based decision making that could challenge conventional wisdom. Yet, where we have pressure to measure and we do so too simplistically and do not take into account variables that we should, measurement serves to strengthen conventional wisdom. We focus on batting averages in picking baseball players, and do not to look more deeply and what drives value. The team that can challenge this wisdom gains.
When we look at societal issues, this is bigger. What decisions would we make differently if we relied on measures other than GDP to guide policy making? What if everybody was tracking the wrong thing, and we all jump off a cliff? For years flawed financial risk models gave a green light for the widespread use of collateralized debt obligations and credit default swaps that destabilized the world’s financial systems. The value of the asset losses in the US alone is estimated by the IMF to be greater than $4000 billion. This led us to a state where it was impossible to measure even what is normally easy to measure – the worth of a house or a mortgage.
As our world has become more measured, and the pressure to measure performance and impact has shifted beyond business to all facets of society, we have to be careful to not measure only what what is convenient and standard to measure, but what is important. This is critical where we have had standard ways of measuring success that are very entrenched - like test scores in education. Too often we don't have adequate resources, and we do not always include input from people who can tell us what matters. There are great tools in both the business world and the social change world to address this I look optimistically to the promise of Big Data to help us analyze data to yield insights we may not otherwise expect.
While most of us are neither baseball team managers or Wall Street executives, here are a few ways we can look beyond the conventional and obvious. .
- Think through how you create value, rather than quickly picking indicators based on what others may be measuring (though of course understanding what others are measuring is an important starting point. Mapping value creation goes for both financial value and social value. By doing so you can move to tracking indicators that help you to predict and manage future value.
- Identify and track the levers that can influence value-creation. These ‘drivers’ of value can also change over time, so its important to evaluate what you are measuring. If your indicators don’t help to inform an action to take, then you may want to revisit them. Frontline staff need to derive insights from the data too, not just a distant manager at a future point in time.
- Don’t try to narrow down what you track at the expense of understanding. There is a keen desire among many organizations to have a small scorecard with a very limited number of indicators. This is understandable, but there should still be enough indicators to support your understanding of cause and effect that relate to the goals that your organization is pursuing. Social change in particular is complex.
The baseball pic is Dottie Kamenshek
http://www.britannica.com/EBchecked/topic/310578/Dorothy-Kamenshek