When CBC’s Fifth Estate program launched Rate my Hospital last week, I couldn't help but immediately look up the grade of my hospital across the street. I felt smug that Eagle Ridge wasn't one of the Lower Mainland's batch of terribly ranked hospitals. But should I really sleep better? For the ranking, grades were assigned to 240 acute-care hospitals across Canada based on expert-informed method using five publicly available indicators. The site also allows visitors to rate hospitals in 5 categories - respect, communication, timeliness, cleanliness and whether they would recommend it.
Based on many of the reactions of the powers-that-be in the health systems, I shouldn’t read much into them. Ontario Hospital Association's CEO and president, Pat Campbell told the CBC:
"Hospitals are very large very complex organizations, and to be rating anything on the basis of five measures is incomplete at best and potentially misleading at worst." 1
This recent grading exercise joins the ranks of unsolicited accountability projects that are couched in the public's interest but also seem quite good at generating controversy and media revenue. This includes Maclean's university rankings and MoneySenses’ rankings of charitable organization effectiveness based on fundraising costs and overhead ratios.
Regardless of your ultimate feeling about media driven rankings, there is a lot we can learn from these exercises to think how we can best use performance measurement as an effective tool, both across organizations and within an organization.
The public seeks clear guidance to help them make decisions and wants to be included in the conversation about social value.
While its hard to not think about these ranking exercises only in terms of their potential for publicity and revenue, I think they underline a hunger for accessible and user-friendly information to help when making a decision about education, charitable giving and health care options. Given that within one day of launch, 23,000 people had rated their hospitals, people want to share their perspective and experience, and be part of a conversation about improving health care quality in Canada. Perusing comments on the CBC website reveals one of the most interesting debates about performance measurement I’ve seen.2
Take control of how you communicate your value, or somebody else will for you.
Universities, hospitals and charities do performance measurement and have a wealth of data and insights about how they are doing, and are in the best position to reach out to their stakeholders for feedback and what they want to know. Yet, in the aggregate, there is no leadership to synthesis, analyze and communicate these data in an effective way. Media filled in the gap.
This notion hasn't gone unnoticed by those rated. In response to the Maclean’s University rankings, former University of Toronto President David Naylor wrote in a newspaper op-ed:
"As academics, we devote our careers to ensuring people make important decisions on the basis of good data, analyzed with discipline. But Canadian universities have been complicit, en masse, in supporting a ranking system that has little scientific merit because it reduces everything to a meaningless, average score." 3
The problems of roll up and removing context
Ross Memorial Hospital was assigned a D in the category of death after major surgery, though receiving a B overall. Brian Payne, CEO and president of the hospital took exception about interpreting this indicator out of context. He told the National post that:
"It is unfortunate that by singling out a single metric and assigning it a bad score, the CBC report will undoubtedly raise questions in the public's mind about the quality of our surgical services.”4
He explained that the indicator reflects the significant senior population in the hospital's catchment area and that the surgical team often performs procedures on very elderly patients.
This particular anecdote raises a key problem with presenting a sliver of data no context. While the Fifth Estate notes that the rankings are meant as a starting point for the public to ask questions, how many people will stop at the letter grade in their judgment? Will people have the time and dig further. How many will just pass on the letter grade to family and friends, severing the grade even further from the context?
Our quest for simple, aggregate measures have to be balanced with ensuring that that we can reasonably interpret what we are seeing. This is particularly true for reducing the value of charitable organizations to the overhead ratio and fundraising expenses, given the huge range of activites and missions that these organizations are engaged in. See Dan Pallotta’ TED talk for a passionate stand on measuring the value of work by non-profits.
Demonstrating Value's Measurement Philosophy
Unsolicited ranking projects highlight key issues in measurement that have guided Demonstrating Value's measurement philosophy and framework. First of all, organizations and sectors need to take control of their measurement while being transarent, and to be proactive in developing not only data collection systems, but means to communicate and apply the data. Secondly, organizations should open the conversation to all its stakeholders (including the public at large, where appropriate) to evaluate its performance. People want to be asked. Thirdly, as much as we desire the simplicity that a few indicators, a rolled up indicator, or letter grade can offer, we need to present meaningful narrative and complimentary information to meaningfully interprete and use the data. Performance Snapshots are designed with this objective in mind to easily show what's important, without burying the audience in piles of information.
Rankings Discussed in this Blog:
Notes:
1. “CBC hospital rankings draw condemnation and praise” posted April 10, 2013, CBC New Website [accessed on-line April 23, 2013 www.cbc.ca/news/canada/story/2013/04/10/canada-hospital-rating-reaction-rmh.html]
2.See comments at the bottom of the page: www.cbc.ca/news/canada/story/2013/04/10/canada-hospital-rating-reaction-rmh.html
3. David Naylor "Measuring Up: What University Rankings Do and Don’t Tell Us." Opinion piece, Ottawa Citizen, April 23, 2006.
4. Dave Flaherty and Lindsay Post “Hospitals Slam CBC Report,” The National Post, Monday, April 15, 2013 [accessed on-line April 23, 2013 www.thepost.ca/2013/04/15/hospital-slams-cbc-report]