December 11, 2008

What Numbers Can't Do

Recently I had the interesting experience of reading survey results relating to a subject I actually knew something about. At first blush, the numbers were quite impressive. And then I read a little more closely and discovered that the presentation gave the impression of results that were better than warranted by reality. Since just the "bare numbers" had been reported, important context and nuance were lost. As a result, the story the numbers told was a little misleading.

So how do we restore context, nuance and meaning? And, more importantly, how do we help initiate needed change within our organizations? According to the folks at Anecdote, the answer lies in telling good stories and then listening properly to those stories:

Surveys and metrics can uncover trouble in an organisation, but they usually don't help you identify the reasons for dysfunctions, let alone generate the resolve to springboard people into action. Instead, learn to use stories as listening posts and tap into the emotion to spark action. From time immemorial, stories have contained collective lessons in condensed form. When gathered and examined, stories that are told in your organisation reveal important themes and patterns that in turn indicate effective solutions.

To be clear, I'm not trying to trash quantitative analysis. However, I do believe there are some things that can be communicated best by numbers and other things that can be communicated accurately only through narrative. Be very sure that when you make your choices about what to measure, how to measure and how to report the results, you choose the right tools and methods. If you cut corners here you will compromise your project and, possibly, your credibility. Why risk it?

[Thanks to Stan Garfield for pointing out the Anecdote post.]

2 comments:

Andrew Gent said...

Hi Mary,

Nice post. There is a misperception that "numbers don't lie". But they can be misused or ill-used. Can't say enough about Edward Tufte's books on the subject.

Missing context is one issue. Biased interpretation is equally problematic. I used to say I can do anything with numbers, meaning I could support any argument -- pro or con -- simply by "bending" the numbers to match the cause. One needs to be careful that in one's enthusiasm for a proposal, you aren't providing too much context (or only selected numbers) to strengthen your case.

Mary Abraham said...

Thanks, Andrew. You're absolutely right. When survey results are not reported with integrity we run into trouble. What makes things more difficult is that, as you say, people improperly believe that "numbers don't lie." Consequently, they don't always read numerical reports with the necessary level of skepticism. By contrast, when they hear an anecdote, they assume bias. This is one of battles I bet Shawn Callahan and his colleagues at Anecdote fight every day when trying to change our view of the value of storytelling in the business world.

- Mary