kirstyevidence

Musings on research, international development and other stuff

Policy influence versus evidence-informed policy

22 Comments

Many donors who fund development research seek to measure the policy influence that a given research project (or a group of projects) have had. This is important and valid and the approach has yielded some interesting lessons (see for example here and here). However, it is important to realise that this is not the same as measuring evidence-informed policy.

maizeTo illustrate this, lets imagine a story. A large agricultural company carries out some research that demonstrates that a new strain of genetically modified maize increases yield by 50%. In order to disseminate this research the company organises an all-expenses paid trip to one of their farms in a neighbouring country for a group of MPs who sit on the agricultural select committee. Following this trip, the MPs recommend that the government changes its policy to allow this new strain.

In this case, you can definitely say that the research (or at least the method of disseminating it) has had policy influence. However would you say that policy is evidence-informed? Potentially not. Evidence-informed policy is that which has considered and evaluated a range of evidence. So for example in this case, you would expect the MPs to also have looked at research on the environmental, economic and social impacts. If they have not considered this range of evidence then the policy is not evidence-informed – at least not by my definition.

This may be an extreme case but I think it illustrates the point that finding out that a given piece of research has had impact does not necessarily mean that policy is evidence-informed. It could potentially just mean that those communicating the research have lobbied more effectively than others. If the research was good and the resulting policy change is good for poor people, this can seem like a positive outcome. But the danger is that the policy makers will be equally swayed by the next lobby group who comes along and argues their point effectively – and that group may not have such altruistic motives!

What those of us who support evidence-informed policy would prefer to see is that policy makers (and those who advise them) systematically evaluate the evidence base. We would prefer them to be swayed by the quality of the evidence rather than the charm of the communicator! Communication is important but we must not ‘dumb down’ the role that policy makers can and should play.

Of course these two approaches can go hand in hand. Measuring policy impact of research is an important way for donors to evaluate whether the research they are funding is policy-relevant and communicated effectively. But if we are serious about promoting evidence-informed policy, we need to also look at whether policy makers, and the institutions in which they work, have the capacity and incentives to routinely consider a range of research evidence when they are making policy decisions.

PS Also check out this blog on a similar topic

Advertisements

22 thoughts on “Policy influence versus evidence-informed policy

  1. I like that you’ve said: you’d prefer. This suggests we agree that in reality decisions might not be based on a systematic review. There are some things that are just value based. And I believe that is fine.

    But I agree that it would be ideal if politicians and policymakers had the capacity (themselves or those who work for them) to systematically review the evidence and make a decision. Still, though. what to do, I think, is a matter of values (or incentives, interests, etc.) rather than evidence.

    • I agree with Enrique! It is true for me, to most extent, from what I have experienced in developing countries, that policy making is a complex process based on multiple, sometime unpredictable factors. Evidence is one of that process but evidence needs to be strong, simple and presented in a systematic and consistent way to be an influential factor. Otherwise, evidence may be used for different purposes (e.g., a political/policy game). This may not be the case in developed countries.

      I think informing policy may be one step in the process of influencing policy. There may be different steps or levels of impact that evidence (s) may have on on policy (e.g., informed, advocated, changed).

      Quynh

  2. I absolutely agree Enrique . As a supporter of evidence-informed policy making what I hope for is that policy makers will access and understand the evidence – but I don’t think that evidence can or should tell them what they ought to do. As you point out, those decisions are based on multiple factors. This means that two policy makers could make completely different policy decisions but if both had understood the evidence base on the topic first, I would argue that both are evidence-informed.

  3. Thanks for drawing attention to this, Kirsty. Looking forward to your upcoming posts!
    I recently addressed the seeming influence-evidence conflation (http://www.mwananchi-africa.org/storage/126565_ebpdn%20synthesis-1.pdf), arguing that in African contexts thevassumption that any CSO inevitably argues on the basis of – and provides – evidence-based (or informed!) arguments and thus should be supported to influence policy (shout louder) could promote individuals/groups rather than evidence-based (or informed!) discussion more generally. It also serves to hinder a greater understanding of ‘evidence’ and ‘research’: something becomes ‘evidence’ on account of its influence rather than vice-versa.

    The above does, however, rest upon a belief that it is possible to have an evidence-based (or informed) discussion free from the politics of research uptake. This is a tricky one, and for now I can only suggest that we talk in degrees rather than binary opposition……

    Glad to see this idea gaining currency!

  4. Perhaps if I may pose a questions: Can we considered changing the policy debate matrix as EIPM

  5. And not just dumb down but also undermine. By seeking policy change donors, NGOs, think tanks and now consultancies often bypass the media, professional associations, parliament, parties, etc. In other words, if they can find a direct link to the decision maker they use it. This undermines the capacity of a society to make more informed decisions in the future. This is why we put together a book on think tanks and political parties in Latin America in 2009 (http://www.idea.int/publications/thinking_politics/index.cfm) and lots of other studies that look into think tanks in a wider political context. You can find all these studies in this blog on the political economy on research uptake in which I presented a series of papers by Emma Broadbent: http://wp.me/pYCOD-sQ in which she argues the same thing as you: these are not the same thing.

    The paper on Zambia is particularly relevant for your example: http://www.mwananchi-africa.org/storage/126565_ebpdn_Zambia.pdf

    This point you are making has implications on what we expect from think tanks. Should funders be satisfied with think tanks getting an issue or idea or proposal on to the public agenda (and keeping it there)? Or should they demand that a policy is actually changed? I feel that the former is more the role of a think tank. Policy change expectations feel more like something that is more appropriate for a lobby group.

    In a way tools like the RAPID Outcome Mapping Approach can give the impression that this targeted and direct influencing is more desirable. But in fact it shouldn’t. Think tanks must find and play their role in society; and support the roles that other institutions should play.

  6. Great post Kirsty – the issue is so clear explained by you! I totally agree – on the difference between policy influence vs. EIPM, and also with you, Enrique, on the role of other factors influencing policy, such as values, interests, experience, external pressures, etc. I think that all these factors produce positive/negative incentives to policy makers when they choose how to formulate policies. An issue is, how to make the use of research-evidence appealing to policy makers – in other words, how to make the use of evidence an interest of policy makers? I honestly struggle (and I suspect I am not alone!) to find an answer, but I have a gut feeling now – maybe one way (the only?) is to work to get EIPM as a political value (in its highest conception, not negotiable), of (part of) the society – that is, it is right to work to develop the capacity of policy makers/influencers to understand and use research-evidence, but this will work only when part of the constituencies will ask for it – I suspect this is already happening somewhere. For example, I doubt that the existence of the POST (the UK Parliamentary Office for Science and Technology) would exist if there wasn’t a significant part of the electoral body fancying an EIPM approach (considering it it a political value). The same can be said about the approach of DFID and its growing emphasis on EIPM, M&E and rigorous IE – that, in my opinion, and in times of crisis, became not only necessary, but also politically appealing. From, this, new gut-ideas on the relationship between democracy and EIPM come natural in me, but this is another part of the story. This was a momentary lapse of (un)reason, does it make any sense? Is there any evidence supporting it?

  7. I define impact in (EIPM) as fact based polices that alleviate poverty, soci-economic problems. My argument is policies can be evidence based but lack impact, there are several cases of INGO funded research in developing countries which are fact based but lack impact even after implementation. EIPM advocates should not only argue for fact based policies but also polices which ultimately lead to development.

  8. Great post and replies, full of reflections we all need to make while working in this field. I think one of Kirsty´s last points is crucial: we should further debate on our efforts to promote government institutions that have the capacity and incentives to routinely consider a range of research evidence when they are making policy decisions. This does not mean to ignore other factors of weight when making policy decisions such as values (as Enrique suggests) or budget constrains, etc. but I do believe that there is a need to think about the collective processes and ways of doing things that we want to promote (for instance, that research and accumulated knolledge is considered in the whole decision making process). Thinking about collective processes is mora attractive to me than just working on an individual or organizational level. That{s why we need to work together becuase chaging processes is a HUGE effort. Not a single person nor organization could claim for this type of impact.

    }Morevoer, what will tell us if we´ve been successful? There have been tons of words produced around how, when and if to evaluate how research influences policy but we need to start from the very beginning first: let´s be honest, clear and reasonable in how we define our vision of success. Why should we leave others (i-.e donors) define this? What would make each of us really happy by the end of our own career story? Do we invest enough energy on this?

  9. Pingback: “A policy brief is a piece of paper. It doesn’t DO anything on its own” « on think tanks

  10. Pingback: Supply and demand in evidence-informed policy « kirstyevidence

  11. Pingback: It’s complicated… or is it? « kirstyevidence

  12. Pingback: Supply and demand in evidence-informed policy – this time with pictures! « kirstyevidence

  13. Pingback: The twelve days of evidence « kirstyevidence

  14. Pingback: Getting started on 'evidence-informed' policy - Research to Action - Research to Action

  15. Pingback: Evidence synthesis – what has it ever done for us? | kirstyevidence

  16. Pingback: Results based aid | kirstyevidence

  17. Pingback: Unintended consequences: When research impact is bad for development | kirstyevidence

  18. Pingback: Top-down versus bottom-up development: where does evidence fit in? | kirstyevidence

  19. Pingback: The Topic Guide on Politics and Ideas: Research within the Policy Process | Politics & Ideas: A Think Net

Leave a Reply (go on, you know you want to...)

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s