kirstyevidence

Musings on research, international development and other stuff


8 Comments

Unintended consequences: When research impact is bad for development

Development research donors are obsessed with achieving research impact and researchers themselves are feeling increasingly pressurised to prioritise communication and influence over academic quality.

To understand how we have arrived at this situation, let’s consider a little story…

Let’s imagine around 20 years ago an advisor in an (entirely hypothetical) international development agency. He is feeling rather depressed – and the reason for this is that despite the massive amount of money that they are putting into international development efforts, it still feels like a Sisyphean task. He is well aware that poverty and suffering are rife in the world and he wonders what on earth to do. Luckily this advisor is sensible and realises that what is needed is some research to understand better the contexts in which they are working and to find out what works.

Fast-forward 10 or so years and the advisor is not much happier. The problem is that lots of money has been invested in research but it seems to just remain on the shelf and isn’t making a significant impact on development. And observing this, the advisor decides that we need to get better at promoting and pushing out the research findings. Thus (more or less!) was born a veritable industry of research communication and impact. Knowledge-sharing portals were established, researchers were encouraged to get out there and meet with decision makers to ensure their findings were taken into consideration, a thousand toolkits on research communications were developed and a flurry of research activity researching ‘research communication’ was initiated.

dfid advisorBut what might be the unintended consequences of this shift in priorities? I would like to outline three case studies which demonstrate why the push for research impact is not always good for development.

First let’s look at a few research papers seeking to answer an important question in development: does decentralisation improve provision of public services. If you were to look at this paper, or this one or even this one, you might draw the conclusion that decentralisation is a bad thing. And if the authors of those papers had been incentivised to achieve impact, they might have gone out to policy makers and lobbied them not to consider decentralisation. However, a rigorous review of the literature which considered the body of evidence found that, on average, high quality research studies on decentralisation demonstrate that it is good for service provision. A similar situation can be found for interventions such as microfinance or Community Driven Development – lots of relatively poor quality studies saying they are good, but high quality evidence synthesis demonstrating that overall they don’t fulfil their promise.

My second example comes from a programme I was involved in a few years ago which aimed to bring researchers and policy makers together. Such schemes are very popular with donors since they appear to be a tangible way to facilitate research communication to policy makers. An evaluation of this scheme was carried out and one of the ‘impacts’ it reported on was that one policy maker had pledged to increase funding in the research institute of one of the researchers involved in the scheme. Now this may have been a good impact for the researcher in question – but I would need to be convinced that investment in that particular research institution happened to be the best way for that policy maker to contribute to development.

My final example is on a larger scale. Researchers played a big role in advocating for increased access to anti-HIV drugs, particularly in Africa. The outcome of this is that millions more people now have access to those drugs, and on the surface of it that seems to be a wholly wonderful thing. But there is an opportunity cost in investment in any health intervention – and some have argued that more benefit could be achieved for the public if funds in some countries were rebalanced towards other health problems. They argue that people are dying from cheaply preventable diseases because so much funding has been diverted to HIV. It is for this reason we have NICE in the UK to evaluate the cost-effectiveness of new treatments.

What these cases have in common is that in each case I feel it would be preferable for decision makers to consider the full body of evidence rather than being influenced by one research paper, researcher or research movement. Of course I recognise that this is a highly complicated situation. I have chosen three cases to make a point but there will be many more cases where researchers have influenced policy on the basis of single research studies and achieved competely positive impacts. I can also understand that a real worry for people who have just spent years trying to encourage researchers to communicate better is that the issues I outline here could cause people to give up on all their efforts and go back to their cloistered academic existence. And in any case, even if pushing for impact were always a bad thing, publically funded donors would still need to have some way to demonstrate to tax payers that their investments in research were having positive effects.

So in the end, my advice is something of a compromise. Most importantly, I think researchers should make sure they are answering important questions, using the methods most suitable to the question. I would also encourage them to communicate their findings in the context of the body of research. Meanwhile, I would urge donors to continue to support research synthesis – to complement their investments in primary research. And to support policy making processes which include consideration of bodies of research.

Advertisements


Leave a comment

If he’s going to rest in peace, we might need to stop squabbling

20070907191032!Nelson_MandelaThe media coverage of Nelson Mandela’s passing has provided us with lots of opportunities to remind ourselves of his wisdom and kindness. But, unfortunately, it has also revealed some rather unedifying behaviour which makes me wonder how much we have really learnt from this great man.

Within a few hours of his death, alongside messages of shock and admiration, there was a plethora of shrill messages that person/group X did not deserve to honour him or that person/group Y was not honouring him correctly or that person/group Z was not honouring him sufficiently.

I mean, just to remind you, this man forgave the people who sent him to prison for twenty-seven years! But in the midst of our admiration for him we proved incapable of forgiving people who posted ill-judged facebook messages.

And lest you think this is a tirade against all the angry people out there, I should add that this episode made me reflect on my own tendency to judge and to hold grudges. I have been known to enter into a rage because my water company failed to fix a broken pipe for a few months, I can enter a dark mood because someone slights me on twitter and I have seriously considered how to plot the downfall of a colleague who stayed in a meeting room beyond her allotted timeslot.

Twenty-seven years people, twenty-seven years.

So, my resolution, and one which I welcome you to join me in, is to try to remember and honour Mandela by trying to be a just a little more kind and a little more forgiving to my fellow humans. To attempt to not assume that others are out to get me and to remember that most people are trying their best.

I’m not sure how long I’ll succeed – but I reckon there can be no harm in trying – so go on, fire some aggressive comments at me and watch for my zen-like reaction.


2 Comments

Dealing with disagreement

At a training workshop I attended this week the facilitator argued that while being able to make a logical argument is a vital skill in life, we can all appreciate its limits by considering, when was the last time we won a logical argument with our partners…

chihuahua

This got me thinking about the alternatives –  when logical argument is not enough to overcome disagreement, what options do we have? I have come up with a few below – but I want to preface them by saying they are not offered from a place of expertise but rather from my ongoing experiences of learning the hard way! So apologies if they seem really obvious to you but anyway, here are my options…

Option 1: Give up

The biggest lesson, that I keep failing to learn, is that in many cases it is just not worth trying to argue with someone. If they do not have a similar way of seeing the world – and particularly if they do not share your views about evidence and logic – you are just going to waste your time (as illustrated beautifully – albeit with a bit of strong language – here). In this case you have two further options: if they are being mean, proceed immediately to this outstanding advice on dealing with mean people in subversive ways. However, if they are basically harmless, have a cup of tea, and turn the conversation to something less charged.

Option 2: Figure out if you are close in opinion but facing different directions

I have noticed that many of the heated discussions I get into on a professional basis fall into this category. This is where you are both actually arguing with someone (in your head) who has a far more extreme position than the person in front of you – a little like this…

spectrum

A good example of people standing close but facing in different directions was a seminar I did with LSHTM’s Justin Parkhurst (you can view the whole thing here if you are bored!). In essence, I think we were making very similar points in our talks about evidence and policy making (he did express his ideas more eloquently but I made up for it by having better pictures). The main difference was in some of the language we used and the aspects we chose to emphasise. I imagine that Justin, as a social scientist surrounded by pesky scientists who tend to see the world as measurable and absolute, finds himself having to emphasise the social and cultural influences on the creation of ‘knowledge’. I on the other hand work to promote more use of evidence in policy making and therefore, while I ponder the nature of reality and truth as much as the next person, I also spend a lot of time trying to persuade people that, for the purposes of decision-making, it is helpful to consider certain things (that HIV causes AIDS, that homeopathic remedies do not cure malaria, that Chihuahuas are cute…) as scientifically proven. If you can recognise that you are in this category of disagreement (and luckily in the seminar I mention, we very quickly did), it becomes much more easy to figure out the small differences of opinion you have got and discuss them rationally.

Option 3: Learn from them

My boss, whenever faced with a disagreement, likes to consider what the situation looks like from the other person’s perspective. He reckons that even if what another person is saying seems ludicrous to you, to them, it will seem like a logical and internally consistent position and it is worth trying to figure out what that might look like. This learning might at least help you to deal with the person – but it may also help you to realise that there is some merit in the position they are taking. If you can be open-minded enough to properly listen to a conflicting point of view, you can really learn a lot. I have experienced this by reading a whole load of BIG books in development – Dead Aid, The Bottom Billion, The End of Poverty, The Trouble with Africa, White Man’s Burden, Why Nations Fail etc etc. These books present a range of ways of looking at the world – some of which I instinctively disagreed with from the outset – but I really think I have learnt a lot from each book. Of course, the great advantage of trying to learn from other perspectives is that, just occasionally, you might realise that your perspective was (whisper it) wrong! It reminds me of a story I have heard about Gandhi – when asked why he could contradict this week what he had said last week and he is said to have answered – “It’s because this week I know better!”.


2 Comments

Chickens and eggs

I was listening to a radio debate the other day and the host attempted to sum up the trickiness of the argument by saying “ah well, it’s like the age-old question, what came first the chicken or the egg?”. Which kind of annoyed me cus I have always thought the answer to that question is pretty obvious. You see, at some point in history, our understanding of evolutionary biology* tells us that the sequence of events outlined below occurred:

chickens
As you can see, the very first member of the species ‘chicken’ (as defined by the standard criteria of ability to interbreed with other members of the species) was preceeded not only by the egg from which it emerged, but in fact by a whole lot of eggs which hatched to reveal creatures which over time became increasingly genetically similar to chickens. So you see? It’s simple. It’s the egg!

At this point you may be wondering how I am going to draw out some wise lessons from this story and link them to international development. Well, I’m not. It’s been a long day and I just fancied blogging about chickens. Having said that, now you don’t need to waste any more of your brain power pondering this question and so I trust that you will have much more time to grapple with the big questions of global poverty.

*If you want to learn more about evolution I thoroughly recommend this blog by @HollyDunsworth

**The species name for chickens is Gallus gallus. Which is, by the way, really funny if you come from Scotland. Probably not so much for the rest of you.


3 Comments

Dealing with disagreement

At a training workshop I attended this week the facilitator argued that while being able to make a logical argument is a vital skill in life, we can all appreciate its limits by considering, when was the last time we won a logical argument with our partners…

chihuahua

This got me thinking about the alternatives –  when logical argument is not enough to overcome disagreement, what options do we have? I have come up with a few below – but I want to preface them by saying they are not offered from a place of expertise but rather from my ongoing experiences of learning the hard way! So apologies if they seem really obvious to you but anyway, here are my options…

Option 1: Give up

The biggest lesson, that I keep failing to learn, is that in many cases it is just not worth trying to argue with someone. If they do not have a similar way of seeing the world – and particularly if they do not share your views about evidence and logic – you are just going to waste your time (as illustrated beautifully – albeit with a bit of strong language – here). In this case you have two further options: if they are being mean, proceed immediately to this outstanding advice on dealing with mean people in subversive ways. However, if they are basically harmless, have a cup of tea, and turn the conversation to something less charged.

Option 2: Figure out if you are close in opinion but facing different directions

I have noticed that many of the heated discussions I get into on a professional basis fall into this category. This is where you are both actually arguing with someone (in your head) who has a far more extreme position than the person in front of you – a little like this…

spectrum

A good example of people standing close but facing in different directions was a seminar I did with LSHTM’s Justin Parkhurst (you can view the whole thing here if you are bored!). In essence, I think we were making very similar points in our talks about evidence and policy making (he did express his ideas more eloquently but I made up for it by having better pictures). The main difference was in some of the language we used and the aspects we chose to emphasise. I imagine that Justin, as a social scientist surrounded by pesky scientists who tend to see the world as measurable and absolute, finds himself having to emphasise the social and cultural influences on the creation of ‘knowledge’. I on the other hand work to promote more use of evidence in policy making and therefore, while I ponder the nature of reality and truth as much as the next person, I also spend a lot of time trying to persuade people that, for the purposes of decision-making, it is helpful to consider certain things (that HIV causes AIDS, that homeopathic remedies do not cure malaria, that Chihuahuas are cute…) as scientifically proven. If you can recognise that you are in this category of disagreement (and luckily in the seminar I mention, we very quickly did), it becomes much more easy to figure out the small differences of opinion you have got and discuss them rationally.

Option 3: Learn from them

My boss, whenever faced with a disagreement, likes to consider what the situation looks like from the other person’s perspective. He reckons that even if what another person is saying seems ludicrous to you, to them, it will seem like a logical and internally consistent position and it is worth trying to figure out what that might look like. This learning might at least help you to deal with the person – but it may also help you to realise that there is some merit in the position they are taking. If you can be open-minded enough to properly listen to a conflicting point of view, you can really learn a lot. I have experienced this by reading a whole load of BIG books in development – Dead Aid, The Bottom Billion, The End of Poverty, The Trouble with Africa, White Man’s Burden, Why Nations Fail etc etc. These books present a range of ways of looking at the world – some of which I instinctively disagreed with from the outset – but I really think I have learnt a lot from each book. Of course, the great advantage of trying to learn from other perspectives is that, just occasionally, you might realise that your perspective was (whisper it) wrong! It reminds me of a story I have heard about Gandhi – when asked why he could contradict this week what he had said last week and he is said to have answered – “It’s because this week I know better!”.