A year ago, a district level official in Rwanda informed me that the primary school exam pass rates had been augmented upwards by 10-20%. This was to account for the fact that students were now taking the exams in English, not French. My astonishment at this “slight adjustment” was later confirmed by an NGO working with the government and officials in the Ministry of Education itself. Since an account of this augmentation went unnoted in official reports, there was no way to know the data had been manipulated.
I may have been skeptical about statistics before, but this experience quickly cemented it for me. So, since I’ve been working with a lot of stats for my dissertation lately, it’s an issue that is constantly front and center, especially considering the errors and contradictions I continue to find in government reports. One of the side effects (or perhaps benefits) of studying economics is learning just how easily and in how many ways data can be twisted to your own whims. (It really is no wonder Keynes called econometrics “black magic.”)
Ok, I realize I may have already lost your interest.
Statistics is not a particularly thrilling subject for most healthy and happy individuals. But let’s face it, in development we can’t escape from the nitty gritty numbers and, in fact, we need them. Statistics for the extent of civil conflict per year, primary school enrollment rates, GDP, HDI, FDI…These numbers define what we do. NGOs need them to evaluate their outcomes, bureaucrats to create policies, and journalists to tell their stories.
“Math may be the language of the Devil, but statistics proves that reality really is what you make it.”
- Stephen Colbert
There are a number of problems associated with tallying the counts, one of which being the government-initiated manipulation I encountered.
It’s understandable that getting error and hole-free data in the first place is a problematic task in difficult-to-reach areas. It’s costly. Statistics have limitations and some things just can’t be captured or measured well. Moreover, qualitative surveys that measure opinions provide subjective responses that may or may not reflect reality. And figuring out how to measure outcomes from particular initiatives can be tricky or biased (cue RCTs). On top of that, making projections from said data can be wildly inaccurate and impossible or, at minimum, difficult to forecast but necessary to employ as a fundamental tool in policy creation.
But despite all of the good-intentioned attempts at numeric validity, statistics can also be manipulated.
Academics may use only data that bolsters their arguments or models. Governments and NGOs can fudge figures to maintain funding. Surveys may word their questions in ways that lead to the answers they want. So, it is not only what is presented but how it is selectively presented to the public. Their hearts may in the right place to garner support for their respective causes, but the only way to improve is to work with the truth, even if it’s daunting.
As you may have guessed, this post is ultimately not about how I don’t trust statistics. I do, in fact, trust them. I have to—we have to measure somehow, don’t we? But I trust them with a grain of salt and do all I can to make sure they’re as valid as possible. They are, after all, a limited tool like everything else.
I always start by verifying my (hopefully legitimate) sources and cross-checking the numbers with others. Some governments don’t even have the capacity to collect accurate data and compile a valid GDP measure, so checking with outside sources is often better. I also watch out for studies with policy conclusions based on mathematical models or advanced statistics if I haven’t had the training to dissect them. In that case, I instead try to learn from the analyses of those who can. Many studies claim that one thing causes another without actually providing any proof that the two effects are more than simply related. And different statistics that describe what seems to be the same thing can provide opposing results—for example, poverty can decrease by a certain percentage while actually increasing in absolute terms if population has grown. Thus, the choice of what is selectively presented can lead to wildly different conclusions.
Anyone else have some good methods or tricks to make sure you’ve got good numbers?
“If you don’t count, you don’t count.”
- Motto of Rwanda’s National Institute of Statistics
While all of this might seem very obvious, tedious, and/or dull, it’s something that is integral to the field of development as well as journalism. Moreover, it definitely wasn’t obvious for me when I first started looking at these issues more closely. We take many things for given when we actually should be more skeptical. It’s tiresome to check, double check, and triple check and much easier to cut corners. But, if you can please forgive the pun since I didn’t invent it: if you don’t count, you don’t count.