I was recently asked for my opinion on the links between two common concepts in development: results-based aid (RBA) and evidence-informed policy making. It isn’t something I had really considered before, but the more I thought about it, the more I came to the conclusion that these concepts are very different – and the fact that they are often considered to be related is a bit of a worry.
RBA (a general term I will use here to cover various different mechanisms for paying for development interventions on the basis of outputs/outcomes, not inputs) is a mechanism which relies on the ability to measure and attribute impact in order to trigger payments. In other words, you make a decision (on whether to pay) based only on robust evidence of impact. As I have argued quite a few times before (e.g. here and here), evidence-informed policy is all about using a wide variety of evidence to inform your decisions – while acknowledging that the evidence will always be incomplete and that many other factors will also influence you. In this sense evidence-informed policy is quite different to RBA because although it concerns making a decisions based on evidence – it implies a much broader scope of evidence.
I am not saying this in order to criticise RBA. I think it can be a really useful tool and I am delighted to see some really innovative thinking about how RBA can be used to drive better development outcomes. There is some great writing from Nancy Birdsall and colleagues here on the topic which I highly recommend taking a look at.
But my concern about RBA is that it is sometimes applied to projects where it is not appropriate or, worse, that in the future projects will only be funded if they are ‘RBA-able’. I would suggest that to determine whether RBA is appropriate for a given intervention, you need to ask yourself the following questions:
1 Do you know what the problem is?
2 Do you know what the solution to the problem is?
3 Are you confident that the supplier/implementor will be free to implement the solution (i.e. that achieval or non-achieval of the outcome is broadly within their control)?
4 Is the supplier/implementor extrinsically motivated (i.e. incentivised by money)?
Where the answer is yes to these questions, RBA may be a good contracting approach since it will help incentivise the supplier to put their effort into achieving the outcomes you are interested in. Examples might include contracting a commercial company to build a bridge (where there is a clear demand for these interventions from local decision makers) or providing funds to a developing country government for achieving certain measurable health outcomes.
However, I am sure it has occurred to you that many development projects do not fit this mold.
Let me give an example. Some years ago I was involved in a project to support the use of research evidence in the parliament of a country which I will call Zalawia. We recognised that what the staff of the Parliament of Zalawia did not need was more parachuted-in northern experts to give one-off training workshops on irrelevant topics – they needed support in some basic skills (particularly around using computers to find information), ideally delivered by someone who understood the context and could provide long-term support. So, we supported a link between the parliament and one of the national universities. We identified one university lecturer, let’s call him Dr Phunza, who had a real interest in use of evidence and we supported him to develop and set up a capacity building scheme for the parliament. Our support included providing Dr Phunza with intense training and mentoring in effective pedagogy, providing funds for his scheme and helping him to secure buy-in from the head of research and information in the Parliament. A number of meetings and phone calls took place between Dr Phunza and staff in the parliament over many months and eventually a date was set for the first of a series of training sessions in ‘Finding and Using Online Information’. Dr Phunza developed a curriculum for the course and engaged the services of a co-facilitator. However, when the day arrived, none of the parliamentary staff who were expected to turn up did so – at the last minute they had been offered a higher per diem to attend an alternative meeting so they went there.
So, what would have happened if we had been on a result-based contract with our funders? Well essentially, we would have put all our efforts in, taken up a lot of our time and energy, and spent our funds on transport, room hire etc. and yet we would presumably not have been paid since we didn’t achieve the outcome we had planned. I have worked in many policy making insitutions on projects to support use of evidence I can say that the situation described with Zalawia was in no way unusual. In fact if we had been pushed to use a RBA model for that project, given our knowledge of the inherent difficulty of working with Parliaments, our incentive from the outset would have been to set up a project with a much more achievable outcome – even if we knew it would have much less impact.
So let’s go back to my four questions and apply them to this project…
1. Did we know what the problem was? – well yes, I would say we were pretty clear on that.
2 Did we know what the solution to the problem was? – hmm, not really. We had some ideas that were informed by past experiences – but I think that we still had quite a bit to learn. The issue was that there was no real evidence base on ‘what works’ in the setting we were working in so the only way to find out was trial and (quite a lot of) error.
3 Were we free to implement the solution? – absolutely not! We were completely dependent on the whims of the staff and in particular the senior management of the parliament in question.
4 Were we incentivised by money? – no, not really. I was working for a non-profit organisation and Dr Phunza was a University lecturer. If money had been withheld it would just have meant that some of the other activities we were planning would not have been possible. I suspect that I still would have found funds, even if it was from my own pocket, to pay Dr Phunza.
The other thing that is worth saying is that, given how hard both myself and Dr P worked to get the project running, I think we would have found it quite insulting and demotivating to be told that we would only be paid if we were sucessful – it would have seemed rather rude to us to imply that we would have needed financial incentives in order for us to bother trying!
In other words, I don’t think this type of project would be suitable for RBA. There are many risks inherent in funding such a project but a major one is not that the implementer would not bother trying – and thus the risk mitigation strategy of RBA would be unnecessary – and potentially damaging.
Does this mean that I think our donors should have just left us alone to get on with our project? Absolutely not! I am well aware that many development actors spend many years working hard at interventions they truly believe in which are, in fact, pointless or even damaging. So I am not suggesting that donors should just let people do what they like so long as they are well-intentioned. However, I think we need to choose mechanisms for scrutiny and incentivising that fit the particular aims and context of the development programme in question. And where we don’t have good mechanisms to hand, we need to continue to innovate to develop systems that help us achieve the impacts we seek.
UPDATE: After writing this blog I have had quite a few interesting discussions on this topic which have moved my thinking on a bit. In particular, Owen Barder gave some useful comments via twitter. What I took from that discussion (if I understood correctly) was that in the case I gave, RBA could still have been used but that some sort of ‘risk premium’ would have to have been built into the payment schedule – i.e. the donor would have had to have added some extra funds to each payment above and beyond the actuals cost. He also took issue with the fact that I was saying that implementers were not incentivised by money – he said if this were really the case would implementers spend so much time trying to please donors – a fair point! So perhaps combining a risk premium with PBR would ensure that the implementer was still incentivised to deliver by paying on results but would also mean that they were able to mitigate against the risk that one or more milestone was not met? This still leaves me with some unanswered questions – one issue is how do you work out the extent of the risk in new and novel programmes? Another point that an organisation who are on a milestone-based contract is that they find it reduces their opportunity for innovation – they are tied down to delivering certain milestones and if they realise that better development impact could be achieved by doing something they had not predicted at the time the contract was negotiated, this is difficult. So in summary, this is a complicated topic with far more pros and cons that I probably realised when I started writing! But am grateful to people for continuing to educate me!