Musings on research, international development and other stuff


Capacity building rule 3: do stuff to facilitate learning

This rule may sounds so obvious that it is not even worth stating. But it is amazing how many projects which are labeled as capacity building don’t seem to contain any plans to actually support the building of capacity, i.e. learning.

One common mistake is to think that giving funding to an organisation in the south is ‘capacity building’, as if the money will somehow lead to learning through a process of osmosis. There are plenty more ‘capacity building’ schemes which contain activities supposedly to support learning which are so badly designed and implemented that they are very unlikely to achieve their aims. I have sat through a fair number of ‘capacity building’ workshops that were so deathly boring that the only thing I have learnt is how to pass the time until the next tea break.

The sad thing is that there is actually a lot of good knowledge on how people learn and those who run capacity building could benefit massively from understanding it. I am not talking about the pseudoscientific stuff like the practice of teaching according to learning styles – but the more serious study of pedagogy that has demonstrated what practices really support learning – and which ones should be discarded and at an organisational level, there is lots of good learning on how to support organisational development. It is extremely arrogant of us to assume that just because we know about a given topic that we know how to support others to learn about it.

The point is that you don’t need to start from scratch when designing capacity building – get speaking to people who know and go to some courses in pedagogy/training skills/organisational development and your capacity building programme will be dramatically improved.

Go to rule 4 here… or start with the first post in the series here.



How to to look smart to development geeks

xfactorI have been having an amusing and distracting twitter conversation this week about how to look smart in front of the various different tribes of development specialists. Here’s a few tips to instantly up your credibility no matter who you are meeting with…

If you are meeting a social development expert, no matter what the topic, be sure to ask if they have considered it through a ‘gendered lens’.

In meetings with evaluation experts ALWAYs question the credibility of the counterfactual. If that doesn’t work, you can resort to questioning the external validity.

Make social scientists think you are one of them by dropping the word epistemology into any discussion. For example, try opening a sentence with the phrase “Epistemologically speaking,…”  but be sure to practice this beforehand because if you come out with a few too many syllables all your efforts will have been wasted. “Normative” is another good social science word to throw in and is particularly useful for throwing doubt on someone’s opinion while maintaining the facade that you are just upholding objectivity i.e. “hmm… isn’t that a rather normative stance you are taking?”

People from IDS will invariable nod enthusiastically if you say “I think we need to unpack this a little further”;  ODI types will be more impressed by you alluding to political economy analysis and/or complexity theory; and those working for DFID will love you if you mention value for money in every second sentence.

And of course, everybody’s favourites: the economists – it is just too easy to tease them for their impenetrable jargon. There are so many good economist catchphrases that it is hard to know where to start but I particularly liked @otis_read’s suggestion of “wow, interesting project, except for obvious endogeneity problem” and, from @fp2p: Look em in eye & say “I’m not convinced by your elasticities”

Have a great weekend – some slightly more serious blogs coming up next week.


12 principles for Payment by Results – the simplified version


Meanwhile in Oxford…

Stefan Dercon and Paul Clist recently published this excellent short paper outlining 12 principles to consider before using a Payment by Results (PbR) contract for development programmes. But, as pointed out by @hmryder, it is written in quite technical language. You can’t blame the authors – I mean, they are hardcore economists who probably speak that way when they are watching the football. So I have attempted to translate the paper for fellow simple folk – economists do let me know if I have made any mistakes.

Principle 1: PbR involves paying for something after it has been delivered. Therefore it only works if the implementer has enough money in the first place to pay for the work until they are reimbursed.

Principle 2: If you are going to pay based on results, you need to be able to measure the results. If you choose a proxy indicator (i.e. not the final result you are looking for but something that has to change along the way), you need to make sure that changes in your indicator really suggest that the end result will change too.

Principle 3: Some people will game the system by finding ways to make it seem that they have achieved the results when they actually haven’t. Perhaps more worrying is that if you choose the wrong proxy indicator, it might lead people to concentrate too much on trying to achieve that without trying to achieve the actual end result you are looking for.

Principle 4: Donors shouldn’t use PbR just as a way to reduce their risk, for two reasons. Firstly, donors are actually usually much better able to handle risk than implementing partners. This is because donors tend to be funding lots of projects, so if one or two go wrong, they still know they have others that should work. Implementers, on the other hand, may only have one project so they are likely to be really risk averse. The second reason is that the implementer is already likely to be very susceptible to risk and by transferring the additional risk of potential non-payment, you will probably just make them even more risk averse.

Principle 5: If the thing that you want to achieve is essentially the same as the thing the implementer wants to achieve, PbR may not be that useful. PbR should be used to incentivise implementers to do the thing that you want them to do, and you might be wasting effort if they are already fully incentivised to do that thing anyway.

Principle 6: PbR is useful where it is difficult to measure what the implementers are doing (inputting), and therefore you need to measure what they are achieving. If you can easily measure what they are doing, just do that.

Principle 7: PbR works well when achieving the result you are looking for is actually within the control (more or less) of the implementers. It doesn’t work well when there are loads of factors outside the implementers control which will determine whether the result is achieved.

Principle 8: The biggest extra cost of PbR contracts compared to other contracts is the cost of verifying whether results (or a suitable proxy indicator of results) have been achieved.

Principle 9: There is some evidence that trying to incentivise people who are already very motivated to do something by giving them money can actually backfire – they may feel insulted that you think they need to be paid to do something when actually they want to do it because they think it is the right thing. (I wrote about this a bit here).

Principle 10: Donors need to be honest about the practical constraints they are working under and to be aware when these might get in the way of an effective PbR contract.

Principle 11: You can only judge whether your PbR contract has been sucessful by looking to see whether the end result you were aiming for has actually been achieved. Just showing that a proxy indicator has been achieved is not enough.

Principle 12: Remember that PbR is not the only tool in the box for incentivising performance.



Unintended consequences: When research impact is bad for development

Development research donors are obsessed with achieving research impact and researchers themselves are feeling increasingly pressurised to prioritise communication and influence over academic quality.

To understand how we have arrived at this situation, let’s consider a little story…

Let’s imagine around 20 years ago an advisor in an (entirely hypothetical) international development agency. He is feeling rather depressed – and the reason for this is that despite the massive amount of money that they are putting into international development efforts, it still feels like a Sisyphean task. He is well aware that poverty and suffering are rife in the world and he wonders what on earth to do. Luckily this advisor is sensible and realises that what is needed is some research to understand better the contexts in which they are working and to find out what works.

Fast-forward 10 or so years and the advisor is not much happier. The problem is that lots of money has been invested in research but it seems to just remain on the shelf and isn’t making a significant impact on development. And observing this, the advisor decides that we need to get better at promoting and pushing out the research findings. Thus (more or less!) was born a veritable industry of research communication and impact. Knowledge-sharing portals were established, researchers were encouraged to get out there and meet with decision makers to ensure their findings were taken into consideration, a thousand toolkits on research communications were developed and a flurry of research activity researching ‘research communication’ was initiated.

dfid advisorBut what might be the unintended consequences of this shift in priorities? I would like to outline three case studies which demonstrate why the push for research impact is not always good for development.

First let’s look at a few research papers seeking to answer an important question in development: does decentralisation improve provision of public services. If you were to look at this paper, or this one or even this one, you might draw the conclusion that decentralisation is a bad thing. And if the authors of those papers had been incentivised to achieve impact, they might have gone out to policy makers and lobbied them not to consider decentralisation. However, a rigorous review of the literature which considered the body of evidence found that, on average, high quality research studies on decentralisation demonstrate that it is good for service provision. A similar situation can be found for interventions such as microfinance or Community Driven Development – lots of relatively poor quality studies saying they are good, but high quality evidence synthesis demonstrating that overall they don’t fulfil their promise.

My second example comes from a programme I was involved in a few years ago which aimed to bring researchers and policy makers together. Such schemes are very popular with donors since they appear to be a tangible way to facilitate research communication to policy makers. An evaluation of this scheme was carried out and one of the ‘impacts’ it reported on was that one policy maker had pledged to increase funding in the research institute of one of the researchers involved in the scheme. Now this may have been a good impact for the researcher in question – but I would need to be convinced that investment in that particular research institution happened to be the best way for that policy maker to contribute to development.

My final example is on a larger scale. Researchers played a big role in advocating for increased access to anti-HIV drugs, particularly in Africa. The outcome of this is that millions more people now have access to those drugs, and on the surface of it that seems to be a wholly wonderful thing. But there is an opportunity cost in investment in any health intervention – and some have argued that more benefit could be achieved for the public if funds in some countries were rebalanced towards other health problems. They argue that people are dying from cheaply preventable diseases because so much funding has been diverted to HIV. It is for this reason we have NICE in the UK to evaluate the cost-effectiveness of new treatments.

What these cases have in common is that in each case I feel it would be preferable for decision makers to consider the full body of evidence rather than being influenced by one research paper, researcher or research movement. Of course I recognise that this is a highly complicated situation. I have chosen three cases to make a point but there will be many more cases where researchers have influenced policy on the basis of single research studies and achieved competely positive impacts. I can also understand that a real worry for people who have just spent years trying to encourage researchers to communicate better is that the issues I outline here could cause people to give up on all their efforts and go back to their cloistered academic existence. And in any case, even if pushing for impact were always a bad thing, publically funded donors would still need to have some way to demonstrate to tax payers that their investments in research were having positive effects.

So in the end, my advice is something of a compromise. Most importantly, I think researchers should make sure they are answering important questions, using the methods most suitable to the question. I would also encourage them to communicate their findings in the context of the body of research. Meanwhile, I would urge donors to continue to support research synthesis – to complement their investments in primary research. And to support policy making processes which include consideration of bodies of research.


Results based aid

I was recently asked for my opinion on the links between two common concepts in development: results-based aid (RBA) and evidence-informed policy making. It isn’t something I had really considered before, but the more I thought about it, the more I came to the conclusion that these concepts are very different – and the fact that they are often considered to be related is a bit of a worry.

RBA (a general term I will use here to cover various different mechanisms for paying for development interventions on the basis of outputs/outcomes, not inputs) is a mechanism which relies on the ability to measure and attribute impact in order to trigger payments. In other words,  you make a decision (on whether to pay) based only on robust evidence of impact. As I have argued quite a few times before (e.g. here and here), evidence-informed policy is all about using a wide variety of evidence to inform your decisions – while acknowledging that the evidence will always be incomplete and that many other factors will also influence you.  In this sense evidence-informed policy is quite different to RBA because although it concerns making a decisions based on evidence – it implies a much broader scope of evidence.

I am not saying this in order to criticise RBA. I think it can be a really useful tool and I am delighted to see some really innovative thinking about how RBA can be used to drive better development outcomes. There is some great writing from Nancy Birdsall and colleagues here on the topic which I highly recommend taking a look at.

But my concern about RBA is that it is sometimes applied to projects where it is not appropriate or, worse, that in the future projects will only be funded if they are ‘RBA-able’. I would suggest that to determine whether RBA is appropriate for a given intervention, you need to ask yourself the following questions:

1 Do you know what the problem is?

2 Do you know what the solution to the problem is?

3 Are you confident that the supplier/implementor will be free to implement the solution (i.e. that achieval or non-achieval of the outcome is broadly within their control)?

4 Is the supplier/implementor extrinsically motivated (i.e. incentivised by money)?

carrotWhere the answer is yes to these questions, RBA may be a good contracting approach since it will help incentivise the supplier to put their effort into achieving the outcomes you are interested in. Examples might include contracting a commercial company to build a bridge (where there is a clear demand for these interventions from local decision makers) or providing funds to a developing country government for achieving certain measurable health outcomes.

However, I am sure it has occurred to you that many development projects do not fit this mold.

Let me give an example. Some years ago I was involved in a project to support the use of research evidence in the parliament of a country which I will call Zalawia. We recognised that what the staff of the Parliament of Zalawia did not need was more parachuted-in northern experts to give one-off  training workshops on irrelevant topics – they needed support in some basic skills (particularly around using computers to find information), ideally delivered by someone who understood the context and could provide long-term support. So, we supported a link between the parliament and one of the national universities. We identified one university lecturer, let’s call him Dr Phunza, who had a real interest in use of evidence and we supported him to develop and set up a capacity building scheme for the parliament. Our support included providing Dr Phunza with intense training and mentoring in effective pedagogy, providing funds for his scheme and helping him to secure buy-in from the head of research and information in the Parliament. A number of meetings and phone calls took place between Dr Phunza and staff in the parliament over many months and eventually a date was set for the first of a series of training sessions in ‘Finding and Using Online Information’. Dr Phunza developed a curriculum for the course and engaged the services of a co-facilitator. However, when the day arrived, none of the parliamentary staff who were expected to turn up did so – at the last minute they had been offered a higher per diem to attend an alternative meeting so they went there.

So, what would have happened if we had been on a result-based contract with our funders? Well essentially, we would have put all our efforts in, taken up a lot of our time and energy, and spent our funds on transport, room hire etc. and yet we would presumably not have been paid since we didn’t achieve the outcome we had planned. I have worked in many policy making insitutions on projects to support use of evidence I can say that the situation described with Zalawia was in no way unusual. In fact if we had been pushed to use a RBA model for that project, given our knowledge of the inherent difficulty of working with Parliaments, our incentive from the outset would have been to set up a project with a much more achievable outcome – even if we knew it would have much less impact.

So let’s go back to my four questions and apply them to this project…

1. Did we know what the problem was? – well yes, I would say we were pretty clear on that.

2 Did we know what the solution to the problem was? – hmm, not really. We had some ideas that were informed by past experiences – but I think that we still had quite a bit to learn. The issue was that there was no real evidence base on ‘what works’ in the setting we were working in so the only way to find out was trial and (quite a lot of) error.

3 Were we free to implement the solution? – absolutely not! We were completely dependent on the whims of the staff and in particular the senior management of the parliament in question.

4 Were we incentivised by money? – no, not really. I was working for a non-profit organisation and Dr Phunza was a University lecturer. If money had been withheld it would just have meant that some of the other activities we were planning would not have been possible. I suspect that I still would have found funds, even if it was from my own pocket, to pay Dr Phunza.

The other thing that is worth saying is that, given how hard both myself and Dr P worked to get the project running, I think we would have found it quite insulting and demotivating to be told that we would only be paid if we were sucessful – it would have seemed rather rude to us to imply that we would have needed financial incentives in order for us to bother trying!

In other words, I don’t think this type of project would be suitable for RBA. There are many risks inherent in funding such a project but a major one is not that the implementer would not bother trying – and thus the risk mitigation strategy of RBA would be unnecessary – and potentially damaging.

Does this mean that I think our donors should have just left us alone to get on with our project? Absolutely not! I am well aware that many development actors spend many years working hard at interventions they truly believe in which are, in fact, pointless or even damaging. So I am not suggesting that donors should just let people do what they like so long as they are well-intentioned. However, I think we need to choose mechanisms for scrutiny and incentivising that fit the particular aims and context of the development programme in question. And where we don’t have good mechanisms to hand, we need to continue to innovate to develop systems that help us achieve the impacts we seek.

UPDATE: After writing this blog I have had quite a few interesting discussions on this topic which have moved my thinking on a bit.  In particular, Owen Barder gave some useful comments via twitter. What I took from that discussion (if I understood correctly) was that in the case I gave, RBA could still have been used but that some sort of ‘risk premium’ would have to have been built into the payment schedule – i.e. the donor would have had to have added some extra funds to each payment above and beyond the actuals cost. He also took issue with the fact that I was saying that implementers were not incentivised by money – he said if this were really the case would implementers spend so much time trying to please donors – a fair point! So perhaps combining a risk premium with PBR would ensure that the implementer was still incentivised to deliver by paying on results but would also mean that they were able to mitigate against the risk that one or more milestone was not met? This still leaves me with some unanswered questions – one issue is how do you work out the extent of the risk in new and novel programmes? Another point that an organisation who are on a milestone-based contract  is that they find it reduces their opportunity for innovation – they are tied down to delivering certain milestones and if they realise that better development impact could be achieved by doing something they had not predicted at the time the contract was negotiated, this is difficult. So in summary, this is a complicated topic with far more pros and cons that I probably realised when I started writing! But am grateful to people for continuing to educate me!


Nerds without borders – Justin Sandefur

It’s the last in the series of Nerds without Borders but don’t worry, it’s a good one… it’s only the Centre for Global Development’s JUSTIN SANDEFUR! Find him on twitter as @JustinSandefur

I'm not trying to start rumours*, but has anyone ever seen these two men in the same room??

I’m not trying to start rumours*, but has anyone ever seen these two men in the same room??

1. What flavour of nerdy scientist/researcher are you?I”m an economist.  I’m usually reluctant to call myself a scientist, as I have mixed feelings about the physics-envy that infects a lot of the social sciences.  But for the purposes of your blog series on nerds, I’m happy to play the part.  To play up the nerdy part, I guess you could call me an applied micro-econometrician.  I live amongst the sub-species of economists obsessed with teasing out causation from correlations in statistical data.  In the simplest cases (conceptually, not logistically), that means running randomized evaluations of development projects.

By way of education, I spent far too many years studying economics: masters, doctorate, and then the academic purgatory known as a post-doc.  But my training was pretty hands on, which is what made it bearable.  Throughout grad school I worked at Oxford’s Centre for the Study of African Economies, running field projects in Kenya, Tanzania, Ghana, Liberia, and Sierra Leone on a wide range of topics — from education to land rights to poverty measurement.

 2. What do you do now?

I’m a research fellow at the Center for Global Development (CGD) in Washington, D.C.  CGD is a smallish policy think tank.  If most of development economics can be characterized (perhaps unfairly) as giving poor countries unsolicited and often unwelcome policy advice, CGD tries to turn that lens back around on rich countries and analyze their development policies in areas like trade, climate, immigration, security, and of course aid.

But getting to your question about what I actually do on a day to day basis: a lot of my work looks similar to academic research.  The unofficial CGD slogan on the company t-shirts used to be “ending global poverty, one regression at a time.”  So I still  spend a good chunk of my time in front of Stata running regressions and writing papers.

3. What has research got to do with international development?

That’s a question we spend a lot of time wrestling with at CGD.  Observing my colleagues, I can see a few different models at work, and I’m not sure I’d come down in favor of one over the others.

The first is the “solutionism” model, to use a less-than-charitable name.  I think this is the mental model of how research should inform policy that an increasing number of development economists adhere to.  Researchers come up with new ideas and test promising policy proposals to figure out what will work and what won’t.  Once they have a solution, they disseminate those findings to policymakers who will hopefully adopt their solutions.  Rarely is the world so linear in practice, but it’s a great model in theory.

The second approach is much more indirect, but maybe more plausible.  I’ll call it a framing model for lack of a better term. Research provides the big picture narrative and interpretive framework in which development policymakers make decisions.  Dani Rodrik has a fascinating new paper where he makes the argument that research — “the ideas of some long-dead economist”, as Keynes put it — often trumps vested interests by influencing policymakers’ preferences, shaping their view of how the world works and thus the constraints they feel they face, and altering the set of policy options that their advisers offer them.

My third model of what research has to do with development policymaking is borderline cynical: let’s call it a vetting model.  The result of your narrow little research project rarely provides the answer to any actual policy question.  But research builds expertise, and the peer review publication process establishes the credibility of independent scientific experts in a given field.  And that — rather than specific research results — is often what policymakers are looking for, in development and elsewhere.  Someone who knows what they’re talking about, and is well versed in the literature, and whose credentials are beyond dispute, who can come in and provide expert advice.

I moved to DC as a firm believer in the first model.  CGD gradually pulled me toward the second model.  But when I observe the interface between research and development policymaking in this town, I feel like the third model probably has the most empirical support.

4. What have you been up to recently?

Too many things, but let me pick just one.

This week I’m trying to finally finish up a long overdue paper on the role of development aid during the war in Afghanistan, together with my colleagues Charles Kenny and Sarah Dykstra.  We measure changes over time in aid to various Afghan districts, and look for effects on economic development, public opinion (in favor of the Karzai government and/or the Taliban), and ultimately the level of violence as measured by civilian and military casualties.  To make a long story short: we find some modest bust statistically significant economic return to the billions of dollars spent in aid — even though it was targeted to the most violent, least poor areas.  But we see no effects on either public opinion or violence.

Interestingly, changes over time in public opinion and violence move together quite significantly, in line with some of the basic tenets of counterinsurgency warfare.  But as far as we can measure in Afghanistan, development aid has proven fairly ineffective, on average, at affecting those non-economic outcomes.  Even where households are getting richer and are more satisfied with government services, we see no significant change in support for insurgent groups let alone any decline in violence.

5. What advice would you give to other science types who want to work in development?

People will tell you to get some practical experience, to broaden your interests, to develop your non-research skill set, and so on.  Ignore all that.  Development doesn’t need more smooth-talking development policy experts; development needs world-class experts in specific and often very technical fields.  Follow your research interests and immerse yourself in the content.  If you know what you’re talking about, the rest will fall into place.

 6. Tell us something to make us smile?

I don’t think I believe the advice I just offered under the previous question.  Nor have I really followed it.  But I want to believe it, so hopefully that counts for something.

Thanks Justin – and indeed all my wonderful nerds. It’s been so interesting to hear about everyone’s different career paths and views on research and international development. See the rest of them here: part 1, 2, 3, 4, 5 and 6.

*I am


Higher Education – my two (well, actually four) cents

University graduates in the Philippines, via Jensm Flikr

It seems like higher education is having a bit of a ‘moment’ in the development world just now. More people than ever are enrolling for universities and new modes of delivery such as Massive Open Online Courses (usually referred to by the wonderful acronym ‘MOOCs’) have the potential to transform how post-secondary learning takes place. The High Level Panel’s emphasis on data has focussed attention on the need to strengthen in-country analytical capacity (although it seems that not everyone agrees on how this should best be done!) and indeed there is growing recognition that achievement of development goals in all sectors will require a higher education system which is able to deliver knowledge and human capital. Meanwhile DFID has set up a Higher Education Taskforce to consider a future policy position on higher education.

Tying in with this flurry of interest, the Association of Commonwealth Universities will be launching its Beyond 2015 Campaign – asking whether higher education is ready to contribute to future development goals. They are calling for inputs from a range of stakeholders and, since this is one of my (many!) soap-box issues, I thought I would take the opportunity to throw in a few thoughts and suggestions of my own…
1. Don’t get too seduced by ‘technological fix’ arguments.
The argument for higher education is sometimes made on the basis that an increase in research will lead to new and better technologies which will make the world a better place. Now, there’s some truth in this – many of the greatest technological developments have come from academia – however, I think it is also misleadingly simplistic. The changes needed to end poverty are complex, deeply political and unlikely to be ‘fixable’ with technological breakthroughs. And indeed many exciting technological fixes are under-used due to political barriers. I think the major benefit that higher education can give to society is increased human capital. A major part of this is through vocational training – to produce the nurses, doctors, engineers and teachers of the future. But higher education can also increase the ability of people in all professions to investigate, question and think critically. Such skills are crucial to build societies which grapple with seemingly intransigent problems – and demand better response from their governments.
2. Focus on the organisation…
I know that this is not an original point – but it bears repeating. No amount of funding for research or higher education will lead to sustainable change if the institutions providing it are not well set up and managed. This applies to ‘traditional universities’ – but also to new modes of higher education which may not rely on a physical presence. Support for higher education may need to focus on some of the underlying issues which are crucial, but sometimes not sexy enough to get attention! This includes efficient and transparent finance and accounting systems, effective campus bandwidth management, responsive IT support, well-resourced and proactive libraries etc. etc.
3….but don’t forget the individuals!
There has been a gratifying increase in attention on organisational capacity strengthening in recent years. But occasionally this has given individual capacity building schemes – particularly ones which remove participants from their home institutions – a bad name. Don’t get me wrong – my ideal situation would be that we have world-class higher education institutions in developing countries so that future talent can be nurtured there. But while we are getting there, we don’t want to lose the potential  of lots of talented young people who are seeking an excellent education. Plus, the strengthened organisations of tomorrow are going to need well-educated people to staff them. For this reason, my personal view is that well-targeted individual scholarship schemes which enable talented young people to study at a world-class university and ensure that their new-found skills benefit their own country can be a useful part of efforts to strengthen higher education.
4. Figure out links between research and higher education agendas – and avoid turf wars.
Some projects which are funded as ‘research capacity building’ could equally be described as higher education programmes – and vice versa. I am completely comfortable about this so long as the people funding each talk to each other. The two agendas are so intrinsically linked – and there is no lack of work to do – so I hope we can agree to work together on this one.

I am really looking forward to the discussions on higher education over the next few months – and in particular to hearing the findings of DFID’s Task Force. However it will be important that we don’t let the excitement about higher education distract us from the really pressing needs in other areas of education. As I have discussed before, the state of primary and secondary education remains abysmal in far too many parts of the world – and we will need to focus on all sectors of education if we are to achieve the vision set out in the High-Level Panel report.

1 Comment

Improving on systematic reviews

A fast-track to blogging success in the development field is to pick a research approach (RCTs, econometrics, rigorous synthesis, qualitative research etc), ‘reveal’ that there are some drawbacks of said approach, and go on to conclude that research is bad (or at least highly suspect). Whenever I see such an article, it strikes me as a little akin to giving up on our judicial system on the basis that sometimes there are miscarriages of justice:

I mean, clearly, any method of gathering evidence to inform decisions has limitations. And of course the people making decisions will be informed by a whole lot of other factors. But these facts don’t make me want to give up on evidence entirely but rather inspire me to think about how we can reduce the drawbacks of research approaches and/or support people and processes so that evidence is routinely considered as one part of the decision-making process.

So, it was with happiness that I read this new paper from the ODI. The paper gives an overview of both ‘traditional’ literature reviews and systematic reviews and outlines some drawbacks of each. However, rather than taking the approach of declaring both useless, the authors go on to propose an intermediate approach which combines:

“..compliance with the broad systematic review principles (rigour, transparency, replicability) and flexibility to tailor the process towards improving the quality of the overall findings, particularly if time and budgets are constrained”.

What makes this paper particularly useful is that it sets out a clear process with 8 steps which potential authors can follow. They give plenty of detail of how each stage can be carried out and they include a wealth of useful tips (for example, I learnt about the concept of ‘forward-snowballing’ – who knew?). I think that many people find the idea of carrying out a rigorous review quite intimidating and will find this an invaluable guide. I also love the inclusion of a graphical representation of synthesised evidence – as I have mentioned before, I think we need to get more inventive at communicating bodies of evidence.

The authors don’t shy away from discussing the challenges of their proposed approach – with particular attention paid to the difficulties of assessing the ‘strength’ of evidence. I would tend to be slightly more positive about attempting some type of assessment of evidence strength – and I am not sympathetic to the argument that authors are unable to include method sections due to restrictive word count rules (have you seen how long some academic papers from the ODI are?!). Having said that, I do completely agree with the authors that this is the most challenging – and the most political – part of evidence synthesis and that there will always be a degree of subjectivity.

I did think the authors fell slightly into strawman territory when they list how their approach differs from SRs. A few of the differences do not really exist. For example, they mention that meta-analysis is not a useful way to synthesise data for many topics – which is true – but meta-analysis is by no means a necessary part of an SR. I would hazard a guess that most systematic reviews in development research do not use meta-analysis – see here and here for examples. They also imply that SRs do not include grey literature. This is definitely not true – any good SR should include a thorough search strategy which includes grey literature. See for example this guidance from the EPPI centre which states:

“In most approaches to systematic reviewing the aim is to produce a comprehensive and unbiased set of research relevant to the review question. Being comprehensive means that the search strategy attempts to uncover published and unpublished, easily accessible and harder to find reports of research studies.”

I do wonder whether these statements were true about some earlier SRs – for example, perhaps meta-analysis has been used inappropriately in the past, and I am sure that not all SRs (particularly when they were first introduced to the development research field) did a good job of capturing non-journal published material. This might explain the impressions reflected in the paper.

In any case, these are minor quibbles. Overall, I think it’s a good and useful paper, and I do hope that it will stimulate more people to think about how we can synthesise evidence in a way which is as objective as possible but is also practical.


Chapter 2…in which kirstyevidence meets a randomista!

randomistaOk, I concede… They do exist. I know that I have previously suggested that they are fictional but last week, at a meeting of development practitioners and policy makers, I met a real live randomista who believed that randomised controlled trials were the best form of evidence in all cases and that there were no reasons why you should not always do them!

So – they exist and they are a bit bonkers! But what was even more striking to me about that particular meeting was how out of proportion people’s fear of RCTs seems to be compared to the very small number of people who think like my new randomista friend.

In fact, I am starting to get the impression that being anti-RCT is a bit of a badge of honour for those in the development field. A number of times recently, I have heard quite influential development wonks come out with statements about RCTs which would be comical if they weren’t so dangerous. To be clear, while RCTs are no silver bullet, they have generated knowledge which has saved millions of lives. So why on earth does the development sector hate them so much??

accreditationIt seems to me that it’s fueled by genuine fear that ignorant technocrats are going to shut down important development interventions simply because they do not produce rapid, measurable outcomes. This is a legitimate fear – but RCTs are not the culprit. RCTs are simply a tool not a political ideology.

This fear has generated a number of myths about RCTs which continue to circulate around in blogs and conference talks and don’t seem to die, no matter how many times they are shown to be false. Given their tenacity, I am fairly sure that any attempt I make to disprove them will make little difference – but it’s been a while since I have had a blog-based argument about experimental methodologies, so I think I will give it another go…

So, here are a few myths about RCTs…

MYTH 1: RCTs are useful for measuring things which are easy to count or measure objectively. This is why they are useful in medicine. But they are not useful for measuring ‘softer’ things like changes in attitudes/behaviour/perceptions, particularly when there might be a power imbalance between the investigator and the subject.

This is just not true. Many things which RCTs have been used to measure in medicine are perception-based. For example, there is not an objective biochemical marker for how bad a headache is but RCTs can be used to check if the perception of the headache is improved more when you are given an actual medicine compared to when you are given a placebo. The fact that improvement in headaches is subjective – and responsive to the power dynamics at play between doctors and patients – is precisely why any response to a pill needs to be compared with the response to a placebo so that you can get a measure of what the ‘real’ effect is. Changes in perception are particularly affected by the placebo effect, the desire of participants to please the investigator and a host of other biases which is why RCTs are particularly useful in these cases.

MYTH 2: RCTs=quantitative All other research=qualitative.

This is a common misunderstanding – in fact there is a great deal of quantitative research which is not experimental. RCTs are the most well-known variety of experimental approaches – all this means is that they set up a randomly assigned control group and compare the response in the group which get their actual treatment to the response in the control group. You can also get quasi-experimental approaches – this simply means is that there is a control group but it is not randomly assigned. Any other designs are called observational. These do include qualitative approaches but they also include quantitative research – for example econometric analysis.

MYTH 3: People who support evidence-informed policy believe that RCTs are the gold standard in research.

NOT TRUE (well ok, it may be true for my new randomista friend but it is certainly not a common belief in the circles I move in!). It is true that if you want to find out IF an intervention works, the most rigorous way to find out is to use an experimental approach. This does not mean that you always need to use an experimental approach – sometimes it would be absurd (hat tip to @knezovjb for that link) since there is no other plausable reason for the outcome and sometimes it is not practical. Observational research approaches are equally important but they are used to find the answers to other questions. For example: How does a particular intervention work or indeed why does it not? What is the distribution of a certain condition in a population? What is the economic and political situation of a given environment? etc etc. Observational approaches, such as before/after comparisons, are not the best way to check if something works simply because humans are so susceptible to bias – you may well find lots of people report that the intervention has benefitted them when there is actually no real affect of the intervention beyond placebo effects/desire to please the investigators.

MYTH 4: Development donors only want to fund RCTs.

I often hear people back up their belief in myth 3 by saying that it is clear that donors mainly believe in RCTs based on the fact that they invest so much money in them. This is just not true! I work at DFID and can say with certainty that the majority of research it funds does NOT use experimental approaches. All the data on what is funded by DFID and many other donors is freely available so if you don’t believe me, look it up (at some point when I get the time, I would like to take on a summer student to do a project looking at the data…). Similarly if you look at the evidence which is used in DFID business cases (again all freely available online) the majority is NOT experimental evidence. It is true that there are some bodies which are set up to fund experimental approaches but just as the fact that the Wellcome Trust only funds medical research does not mean that it thinks that agricultural research is less important, the existence of funders of experimental approaches does not in itself mean that there is a grand conspiracy to not fund other research. A variation on this myth is when people have had funding requests for observational research turned down by a development funder with the feedback that the approach lacked rigour. This is sometimes interpreted as meaning that the donors only like experimental approach – but this is not true. We desperately need good observational research – but the key word is good. That means being explicit about your methodology, surfacing and discussing potential biases, exploring alternative potential theories of change, considering if what you are measuring really allows you to answer the question you set out etc etc. See here for some great work on improving the rigour of qualitative approaches to impact assessment.

MYTH 5: RCTs are invariably and uniquely unethical.

It has been suggested that RCTs are unethical since they require that the control group is not given the ‘treatment’ intervention (or at least not at the same time as the treatment group). I think this argument is fairly weak since, whenever an intervention is rolled out, there will be people who get it and those who don’t. It has also been argued that it is unethical to expect participants who are not receiving an intervention to give up their time to contribute to someone’s research when they are not getting any direct benefit in return. I do think this is a valid point that needs to be explored – but this problem is in no way unique to RCTs. In fact, most observational research methods rely on people contributing ‘data’ without getting any direct benefit. Any project that is gathering information from vulnerable projects needs to consider these issues carefully and build an appropriate public engagement strategy.


So, I do think it is really important to have discussions on the value of different types of evidence in different contexts and in fact I am pretty much in agreement with a lot of the underlying concerns that the anti-RCT lobby have: I do get worried that a push to demonstrate results can lead donors to focus more on ‘technological fixes’ to problems instead of doing the ‘softer’ research to understand contexts and explore the reasons why the many existing ‘fixes’ have not achieved the impacts we might have hoped for. But I get frustrated that the debate on this subject tends to become overly polarised and is often based more on rhetoric than facts. I strongly agree with this blog which suggests that we should try to understand each other a bit better and have a more constructive discussion on this topic.


The enthalpy of aid

I have decided to continue my theme of exceedingly geeky analogies by today comparing international development to a chemical reaction. I don’t know if you remember when you learnt chemistry at school enthalpybut one of the few things I can remember is drawing these little ‘enthalpy graphs’ that show how reactions proceed. The graph shown here could for example represent setting fire to a very flamable thing – you need to put a bit of energy in at the beginning, but then the overall you get more energy out than you put in. The little bit of energy you put in at the beginning is called the activation energy.

I think that the very best international development projects are similar to this activation energy. They are projects which inject a little bit of additional funds, know-how, coordination or whatever to people and organisations who already have huge potential to make a difference. This is why I really like the new advertising campaign from Oxfam America – it highlights how relatively small amounts of investment can be enough to support people to really make a difference. It makes a welcome change to the dominant narrative in international development fundraising campaigns – think b-list celebrity rowing down a remote African river, giving out bed-nets to the poor (but happy) Africans. The Oxfam campaign, in contrast, highlights people living and working in developing countries who are working day in day out to make a difference in their community – and it demonstrates that the contribution that external actors can play can be important but is relatively minor.

I think many capacity building programmes would do well to consider the ‘activation energy’ phenomena too. The fact is that capacity buildng programmes – whether they involve mentoring, training, organisational development or whatever – generally only make a marginal change. As an example, lets say that capacity building will lead to a 5% increase in ‘capacity’ to do something (yes I know you can’t measure capacity as a percentage but bear with me…!). You can see that if you offer that programme to people who have 30% existing capacity, it will increase their capacity but they still won’t really be able to do much. But if you are able to work with people who already have 95% capacity, your programme could be just the little bit extra input that is needed to make a real difference.

There are some ethical concerns about this approach – some would say that it is unfair to offer support to those who are already doing relatively well. I think this is a fair comment but I also think that we need to be really honest about how much a given project can ever achieve. If it can only ever offer relatively minor increases in effectiveness, our only choice is to provide this to those who could make use of that to make a real difference. If we aim to support those who are far less able to improve their own situation and those of others then we may need to rethink our project.

Of course this approach completely relies on being able to identify those who have the existing potential – the people and organisations who are at 95%. And unsurprisingly, this is easier said and done! But maybe that is another blog post…