Members can sign in here.

Post

Why I don’t trust statistics

Published 18th June 2010 - 16 comments - 8629 views -

A year ago, a district level official in Rwanda informed me that the primary school exam pass rates had been augmented upwards by 10-20%.  This was to account for the fact that students were now taking the exams in English, not French.  My astonishment at this “slight adjustment” was later confirmed by an NGO working with the government and officials in the Ministry of Education itself.  Since an account of this augmentation went unnoted in official reports, there was no way to know the data had been manipulated.


I may have been skeptical about statistics before, but this experience quickly cemented it for me.  So, since I’ve been working with a lot of stats for my dissertation lately, it’s an issue that is constantly front and center, especially considering the errors and contradictions I continue to find in government reports.  One of the side effects (or perhaps benefits) of studying economics is learning just how easily and in how many ways data can be twisted to your own whims.  (It really is no wonder Keynes called econometrics “black magic.”) 

Ok, I realize I may have already lost your interest. 

Statistics is not a particularly thrilling subject for most healthy and happy individuals.  But let’s face it, in development we can’t escape from the nitty gritty numbers and, in fact, we need them.  Statistics for the extent of civil conflict per year, primary school enrollment rates, GDP, HDI, FDI…These numbers define what we do.  NGOs need them to evaluate their outcomes, bureaucrats to create policies, and journalists to tell their stories.

 

Math may be the language of the Devil, but statistics proves that reality really is what you make it.”

-          Stephen Colbert

There are a number of problems associated with tallying the counts, one of which being the government-initiated manipulation I encountered.   

It’s understandable that getting error and hole-free data in the first place is a problematic task in difficult-to-reach areas.  It’s costly.  Statistics have limitations and some things just can’t be captured or measured well.  Moreover, qualitative surveys that measure opinions provide subjective responses that may or may not reflect reality.  And figuring out how to measure outcomes from particular initiatives can be tricky or biased (cue RCTs). On top of that, making projections from said data can be wildly inaccurate and impossible or, at minimum, difficult to forecast but necessary to employ as a fundamental tool in policy creation.

But despite all of the good-intentioned attempts at numeric validity, statistics can also be manipulated. 

Academics may use only data that bolsters their arguments or models. Governments and NGOs can fudge figures to maintain funding.  Surveys may word their questions in ways that lead to the answers they want.  So, it is not only what is presented but how it is selectively presented to the public.  Their hearts may in the right place to garner support for their respective causes, but the only way to improve is to work with the truth, even if it’s daunting. 

 

I lied.

As you may have guessed, this post is ultimately not about how I don’t trust statistics.  I do, in fact, trust them.  I have to—we have to measure somehow, don’t we?  But I trust them with a grain of salt and do all I can to make sure they’re as valid as possible.  They are, after all, a limited tool like everything else.

I always start by verifying my (hopefully legitimate) sources and cross-checking the numbers with others.  Some governments don’t even have the capacity to collect accurate data and compile a valid GDP measure, so checking with outside sources is often better.  I also watch out for studies with policy conclusions based on mathematical models or advanced statistics if I haven’t had the training to dissect them.  In that case, I instead try to learn from the analyses of those who can.   Many studies claim that one thing causes another without actually providing any proof that the two effects are more than simply related.  And different statistics that describe what seems to be the same thing can provide opposing results—for example, poverty can decrease by a certain percentage while actually increasing in absolute terms if population has grown.  Thus, the choice of what is selectively presented can lead to wildly different conclusions. 

Anyone else have some good methods or tricks to make sure you’ve got good numbers?

 

“If you don’t count, you don’t count.”

-          Motto of Rwanda’s National Institute of Statistics

While all of this might seem very obvious, tedious, and/or dull, it’s something that is integral to the field of development as well as journalism.   Moreover, it definitely wasn’t obvious for me when I first started looking at these issues more closely.  We take many things for given when we actually should be more skeptical.  It’s tiresome to check, double check, and triple check and much easier to cut corners.  But, if you can please forgive the pun since I didn’t invent it: if you don’t count, you don’t count.

 



Comments

  • Giedre Steikunaite on 18th June 2010:

    It is indeed tiresome to check, double check, and then check once again. Especially when it has to do with numbers and formulas. wink This realization that everything might have been manipulated feeds to insecurity of trust. Statistics don’t tell the whole story, but they might be a frame, like a spine, for it. We do indeed take many things for given, as you say, Maria. It would be really relaxing if we had no reason to doubt at least some things in life…


  • Johan Knols on 18th June 2010:

    Hello Maria,

    Interesting post.
    I think that numbers should be used to just give us a rough idea about things. It doesn’t matter if someone earns $1,50 a day or $1,80 (a 20% increase!) because the person will still be poor.
    I remember a census in Botswana in which I was counted three times (although I told them the second time already that I had been counted).
    And how about the oil-spill in the Gulf. Does it really matter if 100.000 liters a day are spilled or 1 million? The fact is that we have a serious problem there and downgrading numbers won’t make the water any cleaner.
    So yes, how important figures are is a good question.


  • Giedre Steikunaite on 18th June 2010:

    Why were you counted three times, Johan? Was it just a mistake or did they need to have more people on the census for some reason?


  • Johan Knols on 18th June 2010:

    They didn’t have enough people so it took a long time to count. They counted me a second and third time while being on safari with guests. They even insisted counting the foreigners that would leave 5 days later!
    One of the things that made me laugh in Africa….


  • Giedre Steikunaite on 18th June 2010:

    OK, so now we know there are three Johan Knolses in Botswana. wink

    Responding to your comment on statistics above, I agree, numbers should give a rough idea of what’s going on, leaving all the details to those who get paid to analyse them. It annoys me when some know-alls undermine important issues on the basis of “wrong numbers”, as in “you said the spill was bad because of 3,000 gallons, but it was actually 2,500 gallons so your argument makes no sense.” This is a fictitious situation, but you get my point.


  • Johan Knols on 18th June 2010:

    @Giedre,

    The other two will participate in Th!nk4.

    I do get your point. Numbers are also a great way of taking the attention of the main issues at hand. “20.000 lions left? Wow, that is a lot!”. But not if you knew that there were more than 200.000 some fifty years back.


  • Iris Cecilia Gonzales on 19th June 2010:

    Interesting post Maria. I think economic journalists also play a major role in making the people understand statistics well. Or sifting through it in the first place.


  • Maria Kuecken on 20th June 2010:

    @Giedre: Agreed, for sure.  It is the misuse and abuse of statistics that really is the problem.  It’s absolutely necessary to have measures to justify policies and make sure appropriate actions are being taken, but it’s easy to use them incorrectly for biased political means.  Once a representative in US Congress took the amount of funding in the stimulus targeted for job creation and divided by the number of jobs created, concluding that the US paid a godawful price per job.  And this is of course, not how it works at all but he tried to use that to justify his opposition.

    @Johan: Haha, thank you—that’s a perfectly illustrative example!  Stats are just one part of the picture.  I definitely think they’re an extremely important part, but when they are used for distraction or quibbling in the face of obvious problems then they become a hindrance and not a help.

    @Iris:  Definitely! I think they’re vital to really getting the main ideas across in an understandable, fast (and hopefully accurate) way by synthesizing all of the info.


  • Sylwia Presley on 24th June 2010:

    I hate stats but I work with them on a daily basis and had to realize that yes, they do matter. We want to see numbers and measure our achievements in them. We need to set up KPI’s and check results. It is crucial to identify needs, given aid as well as actual results.
    ‘if you don’t count, you don’t count.’ - as a linguist and a creative person I hate to say it - but I like this sentence;)


  • Marianne Diaz on 24th June 2010:

    This isa very important subject and anyone interested in development can’t overlook it. Here in Venezuela, it’s very hard to do any serious research because in some very important areas, (such as interpersonal violence and murders by firearms) there aren’t statistics at all. And obviously, you can tell there is a problem because you live it every day, but it’s not serious research if you can’t tell if those affected by violence, for instance, are the 2% or the 20% of the entire population.
    I, as a humanist, have a love/hate relationship with numbers, but I loved this post.


  • Maria Kuecken on 25th June 2010:

    @Sylwia: Absolutely, they are crucial as benchmarks—before, during, and after. And the motto always makes me smile!

    @Marianne: One of my good friends and classmates is from Venezuela and in the past has come across many research problems from the lack of credible data (or data itself). What do you usually do to try to get around that limitation?


  • Marianne Diaz on 25th June 2010:

    @Maria, a long answer. Well, I always start checking if the National Institute for Statistics has data. Sometimes the data is there, but it isn’t compiled as I need it, so I skip the graphics and go straight to -ugh- the long, long, .xls sheets where the rough data is, and in ocassions I have to draw my own statistics from there. A long work.
    When there is no official data, there is usually unofficial data that’s been compiled by some ONG or something similar. In regard to the lack of trust, I usually try to compare the official figures with the ONG figures with the, say, UN or Amnesty or ILO figures for the matter. And in a few occasions, I’ve had to make a poll or something like that in order to draw at least some data in fields where there isn’t any.
    Yes, a long, exhausting work.


  • Hieke van der Vaart on 25th June 2010:

    Hi Maria,
    Very interesting topic! Because how else than with numbers can one measure the progress of the MDG’s? I really wonder, for example, how the development of entire populations is measured, when, for example, no one knows exactly how many people live in metropoles such as Mexico City or well- even in Amsterdam, if you can call that a metropolis smile

    Have a look at the website of this NGO, based in Tanzania. They gather data by using mobile phones, to reach people and issues that are difficult to include in official statistics.

    http://twaweza.org/index.php?i=256


  • Clare Herbert on 03rd July 2010:

    Great post. WE need to be so careful with stats. I read somewhere that looking at the rates of change is wiser than bothering too much with the actual figures. Still, caution is best. The old Churchill line “lies, damn lies and statistics” springs to mind.


  • Maria Kuecken on 03rd July 2010:

    @Marianne: Well, long but comprehensive, at least!  At least that way you can hopefully be more confident about what you find.

    @Hieke: Thanks so much for the link!  It’s a great initiative to work at the local level.

    @Clare: Well, though it depends on what you’re looking at or for, they’re for sure important to keep tabs on.  Hah, yes I love the phrase:
    http://en.wikipedia.org/wiki/Lies,_damned_lies,_and_statistics


Post your comment

  • Remember my personal information

    Notify me of follow-up comments?

    --- Let's see if you are human ---

    Apples grow on what? Add a questionmark to your answer. (6 character(s) required)