Musings on research, international development and other stuff


Holding decision makers to account for evidence use

Evidence-informed policy – it’s a wonderful thing. But just how widespread is it? The ‘Show your workings’ report from the Institute of Government (and collaborators Sense About Science and the Alliance for Useful Evidence) has asked this question and concluded… not very. It states “there [are] few obvious political penalties for failing to base decision on the best available evidence”. I have to say that as a civil servant this rings true. It’s not that people don’t use evidence – actually most civil servants, at least where I work, do. But there are not good systems in place to distinguish between people who have systematically looked at the full body of evidence and appraised its strengths and weaknesses – and those who have referenced a few cherry-picked studies to back up their argument.

cat up tree

Rosie is my actual cat’s name. And she does indeed make many poor life decisions. Incidentally, I named my other cat ‘Mouse’ and now that I am trying to teach my child to identify animals I am wondering just how wise a life decision that was…

The problem for those scrutinising decision making – parliament, audit bodies and, in the case of development, the Independent Commission for Aid Impact – is that if you are not a topic expert it can be quite hard to judge whether the picture of evidence presented in a policy document does represent an impartial assessment of the state of knowledge. The IoG authors realised this was a problem quite early in their quest – and came up with a rather nifty solution. Instead of trying to decide if decisions are based on an unbiased assessment of evidence, they simply looked at how transparent decision makers had been about how they had appraised evidence.

Now, on the evidence supply side there has been some great work to drive up transparency. In the medical field, Ben Goldacre is going all guns blazing after pharmaceutical companies to get them to clean up their act. In international development, registers of evaluations are appearing and healthy debates are emerging on the nature of pre-analysis plans. This is vitally important – if evaluators don’t declare what they are investigating and how, it is far too easy for them to not bother publishing findings which are inconvenient – or to try multiple types of analysis until, by chance, one gives them a more agreeable answer.

But as the report shows, and as others have argued elsewhere, there has been relatively little focus on transparency on the ‘demand’ side. And by overlooking this, I think that we might have been missing a trick. You see, it turns out that the extent to which a policy document explicitly sets out how evidence has been gathered and appraised is a rather good proxy for systematic evidence appraisal. And the IoG’s hypothesis is that if you could hold decision makers to account for their evidence transparency, you could go some way towards improving the systematic use of evidence to inform decision makers.

The report sets out a framework which can be used to assess evidence transparency. As usual, I have a couple of tweaks I would love to see. I think it would be great if the framework included more explicitly an assessment of the search strategy used to gather the initial body of evidence – and perhaps rewarded people for making use of existing rigorous synthesis products such as systematic reviews. But in general, I think it is a great tool and I really hope the IoG et al. are successful in persuading government departments – and crucially those who scrutinise them – to make use of it.



Jaded Aid – the sequel?

I’ve been a little bit delighted to see the publicity that the Jaded Aid card game has been generating (see for example this Foreign Policy write-up). Nothing is more loltastic for development workers that some wryly-observed development humour. As Wayan Vota observes in that FP article, it is affectionate humour; most of us care deeply about the work we are doing. But if we can’t laugh at some of the absurdities of our industries we might go mad (or explode with pomposity).

In this spirit, I’ve been thinking about what other business projects I could crowd-source funding for from the jaded aid generation. I think I have come up with some crackers. Here they are – in ascending order of cynicism 😉

Somehow I suspect that my business ideas wouldn't get me that far on Dragons' Den...

Somehow I suspect that my business ideas wouldn’t get me that far on Dragons’ Den…

1. Many people have questioned aid workers abilities to actually end world poverty – but surely no-one could deny their deep, contextual knowledge of long-haul flights and seedy business hotels. I mean, I don’t know anyone else as good as me at securing the best seat in economy class or blagging my way into business class lounges. So my first idea is to combine this latent travel knowledge with another skill which development workers have – creating online knowledge repositories My one-stop-shop would enable seasoned development workers to mentor and share knowledge with long-haul tourists looking for exotic adventures. Development workers will get the satisfaction of being truly useful. And the boost to the tourist industry may benefit poor countries more than many misguided development projects: win win.
2. Wherever you go in the world, you get ethnic spas based on sanitised versions of indigenous health beliefs. So you get Thai spas with incense, Thai muzak and traditional Thai massages; Indian spas with ladies in saris, treatments inspired by Ayurveda and incense. And, to my surprise, I recently came across a ‘traditional African spa’ with treatments inspired by African traditional medicine carried out to the sound of the Soweto gospel choir. And incense.
People love these spas because it is well known that while people in developing countries may lack wealth, they are rich in indigenous wisdom, charmingly exotic practices… and incense.
So, my proposal is that we give something back to all these developing countries from which we have appropriated our luxury spa treatments. And what better gift than the marvelous indigenous health system of Germany: homeopathy*. I suggest that we set up our german spas across the developing world. Homeopathic massages will have been watered down to the extent that no actual massage is left. Instead, customers will sit in a room with a mulleted German masseuse listening to the relaxing sound of David Hasselhof – and sniffing bratwurst-scented incense. Health impacts will be mediated by the placebo effect – and the huge pitcher of German beer you will be given before leaving.
3. I have long felt a dilemma about gap-year voluntourism projects. On the one hand, I feel that sending under qualified people to carry out projects in poor countries can be patronising, unhelpful and potentially undermining to local economies. Or the other hand, I do think that it is useful for young, impressionable people to have the chance to connect with people from other cultures and (hopefully) to realise that people the world over are just people. I wondered if there is a way to enable this cross-cultural exchange without the patronising well-digging projects. Which is how I came up with the Mzungu** houseboy project. The idea is to link up earnest European gap year students with nouveau riche African families. To be precise we would need to find a particular subset of the newly wealthy who want to show off to their friends and family. The Europeans would get an authentic experience of poverty as a houseboy/girl – secure in the knowledge that they are not falling into the white saviour cliche. And the ostentatious families of Lagos, Nairobi or Kampala get the ultimate status symbol; a European houseboy! What is not to like?!

* A system of alternative medicine/quackery – invented in Germany – where active substances are diluted down to infinitesimally low concentrations.
**The word for a European/Caucasian (or sometimes foreign-resident Africans) in many Bantu languages


International development and the ‘f’ word

I’m not sure if I have mentioned it, but I am kinda into gender equity.

Or, more precisely, I am a card-carrying, misogyny-hating, bra-burning*, don’t-you-dare-tell-me-I-can’t-do-that-just-cus-I-have-a-uterus kind of feminist.

Most of my friends are similarly inclined** and thus, recently, I got into a discussion about how feminism relates to international development. We talked about two facets – how feminist is the international development movement in its actions and how feminist is the international development industry as an employer.

The answer to the first question seems obvious: development is obsessed with gender issues and supporting women and girls. Surely it is more feminist than Caitlin Moran reading the Female Eunuch while chanting suffragette slogans? Well, sometimes. It is true that many of those working on projects for women and girls, regardless of gender, are feminist in thought and action. But my friends and I also noted that projects targeting women are particularly susceptible to the ‘white saviour’ myth; there are some who love the idea of parachuting in to save poor vulnerable women from primitive conditions. This type of rhetoric frequently comes from men, but not always. In fact, it is particularly prevalent in that most un-feminist of publication – the women’s glossy magazine. Inserted between articles about why you should feel inadequate about your body or spend ridiculous amounts of money on your appearance, there is often an article about someone who has gone out to Africa to save the poor vulnerable and helpless women there.

breakout sessionsThis patronising approach is popular with celebrities aiming to show their caring side – but to some extent it can seep into serious development agencies. One of my friends, a gender specialist, described a recent development conference she had attended. She noted that during the tea breaks, there was a good mix of genders but when the breakout sessions started the (mainly male) economists and political scientists went off to discuss the meaty governance issues while the (mainly female) social development and gender specialists were ushered into rooms to discuss more ‘fluffy’ issues. It drove my friend mad. She didn’t think gender should be reduced to ’boutique’ projects about disadvantaged women but that rather there was a need to think about power relationships much more generally. She said she felt like shouting “I want to talk about gender-sensitive tax regimes not about periods (well, not always but I reserve the right to also be allowed to talk about that too but at my choosing!)!”

So what about the development industry as an employer – to what extent does it support, promote and empower women? In my career, I have encountered the odd sexist person, but have generally found that the people and organisations I work with promote gender equity more than seems to be the case in many other industries. Actually, in some cases I have assumed that someone has a problem with women but have later discovered they are just downright rude – in a gender non-specific manner. My other development friends reported a range of experiences on this – some, like me, had not found sexism too much a problem but others had encountered it frequently.

But whether or not people are overtly sexist, there are structural issues in the industry which may disproportionately impede women. I will give a couple of examples.


The point at which the conversation turns to competitive development stories is definitely the time to LEAVE the bar…

Development people are preoccupied with the level of experience that people have ‘in the field’. This is completely justified – I have witnessed the problems you can get when people who have no clue about life in developing countries are managing development programmes, particularly if they don’t have insight into their own lack of knowledge. However, this principle is sometimes used as an excuse for a slightly macho bragging culture and, at times, downright rudeness towards those who are perceived as being less experienced. Unfortunately, people who are more introverted, younger and/or female seem to be disproportionately targeted in this way. And thus I have seen some very unedifying meetings in which a collection of people who have happened to be mainly male have acted in a very discourteous and disrespectful way towards a collection of people who have happened to be mainly female. I suspect these people are not sexist per se – but the combination of their assumptions and their bad manners can still result in gender discrimination. In fact, I suspect that such attitudes probably have a disproportionate effect on other groups as well – including those who come from poorer backgrounds who perhaps don’t have the same level of self-assurance that a lifetime of relative privilege brings with it.

Another issue is that certain groups may genuinely be less able to gather ‘field experience’ – but may have much to offer. Parents with caring responsibilities may not be able to travel overseas at short notice – and, although this is slowly changing, this currently disproportionately affects mothers. Once again, women are not the only group disadvantaged in this way; individuals with physical disabilities may not be able to travel to all locations while people with mental illnesses may struggle with the emotional impact of overseas travel.

None of these issues are insurmountable – but it is important to at least recognise that the industry is set up to favour cis-gender, straight, able-bodied, white, middle/upper class men. By starting with this knowledge, it is possible to consider what actions we can take to improve opportunities for a variety of groups – including women.


*Of course I don’t really burn bras since EU regulations have made them all flame-retardant. They spoil all our fun.

** To be honest, I kind of expect you are too. My view on feminism is summed up by comedian Azis Ansari when he says:

‘If you believe that men and women have equal rights, and then someone asks you if you’re a feminist, you have to say yes. Because that’s how words work. You can’t be like, “Yeah, I’m a doctor who primarily does diseases of the skin.” “Oh, so you’re a dermatologist?” “Oh that’s way too aggressive of a word, not at all, not at all.”’


Race for impact in the age of austerity

I have recently been pondering what the age of austerity means for the development community. One consequence which seems inevitable is increasing scrutiny of how development funds are spent. The principle behind this is hard to argue with; money is limited and it seems both sensible and ethical to make sure that we do as much good as possible with what we have. However, the way in which costs and benefits are assessed could have a big impact on the future development landscape. Already, some organisations are taking the value for money principle to its logical conclusion and trying to assess and rank causes in terms of their ‘bang for your buck’. The Open Philanthropy project has been comparing interventions as diverse as cash transfers, lobbying for criminal justice reform and pandemic prevention, and trying to assess which offers the best investment for philanthropists (fascinating article on this here).

The Copenhagen Consensus project* is trying to do a similar thing for the sustainable development goals; using a mixture of cost-benefit analysis and expert opinion, they are attempting to quantify how much social, economic and environmental return development agencies can get by focussing on different goals. For example, they find that investing a dollar in universal access to contraception will result in an average of $120 of benefit. By contrast, they estimate that investing a dollar in vaccinating against cervical cancer will produce only $3 average return. Looking over the list of interventions and the corresponding estimated returns on investment is fascinating and slightly shocking. A number of high profile development priorities appear to give very low returns while some of the biggest returns correspond to interventions such as trade liberalisation and increased migration which are typically seen as outside the remit of development agencies (good discussion on ‘beyond-aid agenda’ to be found from Owen Barder et al. at CDG e.g. here).

In general, I find the approach of these organisations both brave and important. Of course there needs to be a lot of discussion and scrutiny of the methods before these figures are used to inform policy – for example, I had a brief look at the CC analysis of higher education and found a number of things to quibble with, and I am sure that others would find the same if they examined the analysis of their area of expertise. But the fact that the analysis is difficult does not mean one should not attempt it. I don’t think it is good enough that we continue to invest in interventions just because they are the pet causes of development workers. We owe it both to the tax payers who fund development work and to those living in poverty to do our best to ensure funds are used wisely.

Achieving measurable impacts without doing anything to address root causes

Achieving measurable impacts without doing anything to address root causes

Having said all that, my one note of caution is that there is a danger that these utilitarian approaches inadvertently skew priorities towards what is measurable at the expense of what is most important. Impacts which are most easily measured are often those achieved by solving immediate problems (excellent and nuanced discussion of this from Chris Blattman here). To subvert a well-known saying, it is relatively easy to measure the impact of giving a man a fish, more difficult to measure the impact of teaching a man to fish** and almost impossible to measure, let alone predict in advance, the impact of supporting the local ministry of agriculture to develop its internal capacity to devise and implement policies to support long-term sustainable fishing practices. Analysts in both the Copenhagen Consensus and the Open Philanthropy projects have clearly thought long and hard about this tension and seem to be making good strides towards grappling with it. However, I do worry that the trend within understaffed and highly scrutinised development agencies may be less nuanced.

So what is the solution? Well, firstly development agencies need to balance easy to measure but low impact interventions with tricky to measure but potentially high impact ones. BUT this does not mean that we should give carte blanche to those working on tricky systemic problems to use whatever shoddy approaches they fancy; too many poor development programmes have hidden behind the excuse that it is too complicated to assess them. Just because measuring and attributing impact is difficult does not mean that we can’t do anything to systemstically assess intermediate outcomes and use these to tailor interventions.

To take the example of organisational capacity building – which surely makes up a large chunk of these ‘tricky’ to measure programmes – we need to get serious about understanding what aspects of design and implementation lead to success. We need to investigate the effects of different incentives used in such projects including the thorny issue of per diems/salary supplements (seriously, why is nobody doing good research on this issue??). We need to find out what types of pedagogical approach actually work when it comes to supporting learning and then get rid of all the rubbish training that blights the sector. And we need to think seriously about the extent of local institutional buy-in required for programmes to have a chance of success – and stop naively diving into projects in the hope that the local support will come along later.

In summary, ever-increasing scrutiny of how development funds are spent is probably inevitable. However, if, rather than fearing it, we engage constructively with the discussions, we can ensure that important but tricky objectives continue to be pursued – but also that our approach to achieving them gets better.

* Edit: thanks to tribalstrategies for pointing out that Bjorn Lomborg who runs the Copenhagen Consensus has some controversial views on climate science. This underscores the need for findings from such organisations to be independently and rigorously peer reviewed.

**High five to anyone who now has an Arrested Development song on loop in their head.


Beneficiary feedback: necessary but not sufficient?

One of the things I love about working in DFID is that people take the issue of beneficiary* feedback very seriously. Of course we don’t get it right all the time. But I like to think that the kind of externally designed, top-down, patronising solutions that are such a feature of the worst kind of development interventions (one word: BandAid**) are much less likely to be supported by the likes of DFID these days.

In fact, beneficiary feedback is so central to how we do our work that criticising it in any way can been seen as controversial; some may see it as tantamount to saying you hate poor people! So just to be clear, I think we can all agree that getting feedback from the people you are trying to help is a good thing. But we do need to be careful not to oversell what it can tell us. Here are a couple of notes of caution:

1. Beneficiary feedback may not be sufficient to identify a solution to a problem

problem cakeIt is of course vital to work with potential beneficiaries when designing an intervention to ensure that it actually meets their needs. However, it is worth remembering that what people tell you they need may not match what they will actually benefit from. Think about your own experience – are you always the best placed person to identify the solution to your problems? Of course not – because we don’t know what we don’t know. It is for that reason that you consult with others – friends, doctors, tax advisors etc. to help you navigate your trickiest problems.

I have come across this problem frequently in my work with policy making institutions (from the north and the south) that are trying to make better use of research evidence. Staff often come up with ‘solutions’ which I know from (bitter) experience will never work. For example, I often hear policy making organisations  identify that what they need is a new interactive knowledge-sharing platform – and I have also watched on multiple occasions as such a platform has been set up and has completely flopped because nobody used it.

2. Beneficiary feedback on its own won’t tell you if an intervention has worked

Evaluation methodologies – and in particular experimental and quasi-experimental approaches – have been developed specifically because just asking someone if an intervention has worked is a particularly inaccurate way to judge its effectiveness! Human beings are prone to a whole host of biases – check out this wikipedia entry for more biases than you ever realised existed. Of course, beneficiary feedback can and should form part of an evaluation but you need to be careful about how it is gathered – asking a few people who happen to be available and willing to speak to you is probably not going to give you a particularly accurate overview of user experience. The issue of relying on poorly sampled beneficiary feedback was at the centre of some robust criticisms of the Independent Commission for Aid Impact’s recent review of anti-corruption interventions – see Charles Kenny’s excellent blog on the matter here.

If you are trying to incorporate beneficiary feedback into a rigorous evaluation, a few questions to ask are: Have you used a credible sampling framework to select those you get feedback from? If not, there is a very high chance that you have got a biased sample – like it or not, the type of person who will end up being easily accessible to you as a researcher will tend to be an ‘elite’ in some way. Have you compared responses in your test group with responses from a group which represents a counterfactual situation? If not, you are at high risk of just capturing social desirability bias (i.e. the desire of those interviewed to please the interviewer). If gathering feedback using a translator, are you confident that the translator is accurately translating both what you are asking and the answers you get back? There are plenty of examples of translators who, in a misguided effort to help researchers, put their own ‘spin’ on the questions and/or answers.

Even once you have used a rigorous methodology to collect your beneficiary feedback, it may not be enough to tell the whole story. Getting feedback from people will only ever tell you about their perception of success. In many cases, you will also need to measure some more objective outcome to find out if an intervention has really worked. For example, it is common for people to conclude their capacity building intervention has worked because people report an increase in confidence or skills. But people’s perception of their skills may have little correlation with more objective tests of skill level. Similarly, those implementing behaviour change interventions may want to check if there has been a change in perceptions – but they can only really be deemed successful if an actual change in objectively measured behaviour is observed.


I guess the conclusion to all this is that of course it is important to work with the people you are trying to help both to identify solutions and to evaluate their success. But we also need to make sure that we don’t fetishise beneficiary feedback and as a result ignore the other important tools we have for making evidence-informed decisions.


* I am aware that ‘beneficiary’ is a problematic term for some people. Actually I also don’t love it – it does conjure up a rather paternalistic view of development. However, given that it is so widely used, I am going to stick with it for this blog. Please forgive me.

** I refuse to provide linklove to Bandaid but instead suggest you check out this fabulous Ebola-awareness song featured on the equally fabulous Africaresponds website.


1 Comment

The politics of evidence supply and demand

I have written before about the separate functions of evidence supply and demand. To recap, supply concerns the production and communication of research findings while demand concerns the uptake and usage of evidence. While this model can be a useful way to think about the process of evidence-informed policy making, it has been criticised for being too high level and not really explaining what evidence supply and demand looks like in the real world – and in particular in developing countries.

I was therefore really pleased to see this paper from the CLEAR centre at the University of Witwatersrand which examines in some detail what supply and demand for evidence, in this case specifically evaluation evidence, looks like in five African countries.

What is particularly innovative about this study is that they compare the results of their assessments of evaluation of supply and demand with a political economy analysis and come up with some thought-provoking ideas about how to promote the evidence agenda in different contexts. In particular, they divide their five case study countries into two broad categories and suggest some generalisable rules for how evidence fits in to each.

Developmental patrimonial: the ‘benevolent dictator’

Two of the countries – Ethiopia and Rwanda – they categorise as broadly developmental patrimonial. In these countries, there is strong centralised leadership with little scope for external actors to influence. Perhaps surprisingly, in these countries there is relatively high endogenous demand for evidence; the central governments have a strong incentive to achieve developmental outcomes in order to maintain the government’s legitimacy and therefore, at least in some cases, look for evaluation evidence to inform what they do. These countries also have relatively strong technocratic ministries which may be more able to deal with evidence than those in some other countries. It is important to point out that these countries are not consistently and systematically using research evidence to inform decisions and that in general they are more comfortable with impact evaluation evidence which has clear pre-determined goals rather than evidence which questions values. But there does seem to be some existing demand and perhaps the potential for more in the future. When it comes to supply of evaluations, the picture is less positive: although there are examples of good supply, in general there is a lack of expertise in evaluations, and most evaluations are led by northern experts.

Neopatrimonial: a struggle for power and influence

The other three countries – Malawi, Zambia and Ghana – are categorised as broadly neopatrimonial. These countries are characterised by patronage-based decision making. There are multiple interest groups which are competing for influence and power largely via informal processes. Government ministries are weaker and stated policy may bear little relationship to what actually happens. Furthermore, line ministries are less influenced by Treasury and thus incentives for evidence from treasury are less likely to have an effect. However, the existance of multiple influential groups does mean that there are more diverse potential entry points for evidence to feed into policy discussions. Despite these major differences in demand for evidence, evaluation supply in these countries was remarkably similar to that in developmental patrimonial countries – i.e. some examples of good supply but in general relatively low capacity and reliance on external experts.

I have attempted to summarise the differences between these two categories of countries – as well as the commonalities – are summarised in the table below.

eval tableThere are a couple of key conclusions which I drew from this paper. Firstly, if we are interested in supporting the demand for evidence in a given country, it is vital to understand the political situation to identify entry points where there is potential to make some progress on use of evidence. The second point is that capacity to carry out evaluations remains very low despite a large number of evaluation capacity building initiatives. It will be important to understand whether existing initiatives are heading in the right direction and will produce stronger capacity to carry out evaluations in due course – or whether there is a need to rethink the approach.


Science to the rescue: the big tech transfer myth

This is part 2 of a series of blogs – read part 1 here.

As I mentioned in yesterday’s blog, DFID’s recent lit review on links between science and development started by figuring out how people think science leads to development outcomes. By far the most common justification for investment in research given by developing country policy makers was its expected contribution to economic growth. The Nigerian Science, Technology and Innovation Policy is typical of many in stating:

“Specifically, the new [Science, Technology and Innovation] Policy is designed to provide a strong platform for science, technology and innovation engagements with the private sector for the purpose of promoting sound economic transformation that is citizen centred”

This focus is not likely to surprise anyone who has attended conferences related to science and international development; huge faith is put into Science, Technology and Innovation as drivers of economic development. If evidence is required to back this up, the example of the Asian Tiger economies, which invested in research and subsequently saw unprecedented growth, is frequently cited. For example, this recent communique from the Governments of Ethiopia, Mozambique, Rwanda, Senegal, and Uganda states:

“Inspired by the recent success of the Asian Tigers, We, the African governments, resolve to adopt a strategy that uses strategic investments in science and technology to accelerate Africa’s development into a developed knowledge-based society within one generation.”

If pressed on how research will lead to growth, it is typical to hear statements based broadly on endogenous growth theory: research will lead to new knowledge which will contribute to private sector development which will lead to growth.

So, what does the evidence tell us?

Well for a start, contrary to popular belief, there is little evidence to suggest that public investment in research was a major factor in the economic development of the Asian Tigers. Theories about what did cause this ‘development miracle’ abound but they can be broadly split into two categories. There are those who believe that increased growth was simply due to increased financial investments in the economy – and clearly this camp does not think that public investment in research played much of a role. Then there are those who believe that ‘knowledge’ was a key factor in explaining the rapid growth. At first glance this theory seems consistent with those who advocate for public investment in research to stimulate growth – but when you delve deeper you see that even this latter camp does not suggest that publicly-funded research knowledge was a major driver of growth. In fact, detailed case studies suggest that knowledge which drove growth was accumulated mainly through learning from the processes and technologies of more developed economies and gradually developing in-house R&D capabilities.

A world bank paper from 1999 summarises the findings of case studies from a range of ‘Asian Tiger’ firms as follows:

“. . . firm histories provide details of a complex interactive process in which. . . importers furnished some knowledge of production engineering. . . [Firms] were forced to constantly reduce cost through improving productivity.”

Of course just because public investment in research did not lead to past economic transformations doesn’t mean that it can’t do so in the future. There are many examples of initiatives specifically aimed at stimulating economic growth through publicly-funded research. Perhaps the most well-known – and popular – intervention is the establishment of a ‘science park’. These are generally property developments located near universities which aim to support technology start-ups and companies which ‘spin off’ from the university. They aim to emulate successful technology hubs in the USA, in particular Silicon Valley. The idea is that research in the university will lead to products and technologies which will be commercialised by start-up companies located in the science park.

There has been an explosion of science parks in emerging and developing countries. However, the evidence of their success is less abundant. Beyond a few high profile science parks linked to world-leading universities, there is little evidence that science parks actually succeed in supporting the commercialisation of university-generated research results; studies of science parks from both high-income and middle-income countries demonstrate an almost complete lack of technology transfer from universities to firms. Firms do report some advantages to location in science parks including access to information and informal linkages with academic colleagues. However there is little evidence that firms perform better in science parks than if they were located elsewhere.

In a 2004 article, Professor Stuart Macdonald of the University of Sheffield and Yunfeng Deng of Qingdao National Hi-Tech Industrial Zone describe science parks in developing countries as ‘pure cargo cult’ – aiming to superficially emulate Silicon Valley without creating any of the underlying factors which were necessary for its success. They conclude:

“. . .despite all the enthusiasm, there is little evidence that science parks work as their supporters say, and growing evidence that they do not.”

What could possibly go wrong?

What could possibly go wrong?

Other interventions to drive economic development by supporting technology transfer are not much more promising. Technology transfer offices have been set up in many universities world-wide – however, the evidence shows that the vast majority of such offices work at a loss. In fact, patenting and licensing of academic research knowledge only generates significant income for the very top tier (i.e. the top few percent in global rankings) of universities internationally. A 2005 (paywalled) paper by University of Cape Town academic Dr Anthony Heher, which aims to draw conclusions for developing countries from university licensing data from wealthier economies, concludes:

“Without a well-funded, high quality research system, it is unlikely that a technology transfer programme will make any significant contribution to economic development. It is also not clear that any other country can simply emulate performance of the USA in deriving benefit from technology transfer due to differing social and economic conditions.”

Given this, it seems unlikely that technology transfer from universities is likely to have significant impact on economic development in most developing countries in the short to medium term. In fact, there is evidence that this is unrealistic even in developed countries. A recent article by Times science columnist Matt Ridley concluded that:

“The idea that innovation happens because you put science in one end of the pipe and technology comes out the other end goes back to Francis Bacon, and it is largely wrong.”

There is one silver lining to the evidence on public research and economic growth – there is good evidence that the capacity to use research is an important factor in driving economic growth. And that fact leads neatly on to tomorrow’s blog which will focus on human capital.

Part 3 now available here.


Science to the rescue: does investment in research drive international development?

Meanwhile, at the Kardashians...

Meanwhile, at the Kardashians…

The assertion that research/science* are crucial drivers of development is made so frequently that you could be forgiven for assuming that this is a proven fact. However, having read tens if not hundreds of books, reports and papers about science and international development, I have been struck by the distinct lack of evidence presented to back up the link. I have noticed that authors try to trick us into thinking there is evidence in a couple of ways. Firstly, they often quote eminent, famous or glamorous figures who say that research is crucial for development – and present that as if it is evidence. Alternatively, they will present anecdotes of where research has led to positive changes for poor people and use that to conclude that overall research must be a good thing. This may appear compelling at first, but I can’t help thinking that if I wanted to make the case for gambling, I could probably find quite a number of people whose lives had been improved by winning the lottery – and I am not sure anyone would accept that as good evidence that gambling is overall a great idea.

For me, the question we need to ask is not can research ever lead to good outcomes – but rather, on average, does investment in research lead to better development outcomes than investing an equivalent amount of funds in an alternative intervention?

Luckily for you, DFID has recently produced a literature review which attempted to capture the evidence relating to that question – and even better, I was the lead author. And so, I thought I would write a series of blogs summarising the paper.

The starting point for the lit review was to understand how people think research leads to socio-economic development. To uncover this, research policy documents from developing country governments and major development donors were examined – and informal discussions were held with key actors in the sector. Four common justifications for investing in research to drive development emerged: research was proposed to drive economic growth; to improve human capital; to generate new products and technologies that benefit the poor; and to support evidence-informed policy and practice. Over the next few blogs, I will look at each of these proposed pathways and summarise what the evidence tells us.

It will probably come as no surprise to readers that the answer to the research question I pose above is ‘sometimes’. But the evidence on this topic reveals that the links between research and socioeconomic development are fascinating, complicated and, occasionally, very much at odds with conventional wisdom.

If you want to get subsequent blogs direct to your email just click the ‘follow blog by email’ button on the top right of this post. And if you are too impatient to wait for further blogs, you can go ahead and read the full paper here!

Part 2 is now available here.

*I’ll use these terms interchangeably in this series of blogs


Implementation science: what is it and why should we care?

imp sci pie chart

The 30 participants were mostly members of DFID’s Evidence into Action team plus a few people who follow me on twitter – admittedly not a very rigorous sampling strategy but a useful quick and dirty view!

Last week I attended a day-long symposium on ‘implementation science’ organised by FHI 360. I had been asked by the organisers to give a presentation, and it was only after agreeing that it occurred to me that I really had no idea what implementation science was. It turns out I was not alone – I did a quick survey of colleagues engaged in the evidence-informed policy world and discovered that the majority of them were also unfamiliar with the term (see pie chart). And even when I arrived at the conference full of experts in the field, the first couple of hours were devoted to discussions about what implementation science does and does not include.

To summarise some very in-depth discussions, it seems that there are basically two ways to understand the term.

The definitions that seem most sensible to me describe implementation science as the study of how evidence-informed interventions are put into practice (or not) in real world settings. These definitions indicate that implementation science can only be done after efficacy and effectiveness studies have demonstrated that the intervention can have a positive impact. As @bjweiner (one of the conference speakers) said, implementation science aims to discover ‘evidence-informed implementation strategies for evidence-informed interventions’.

A second category of definitions take a much broader view of implementation science. These definitions include a wide variety of additional types of research including impact evaluations, behaviour change research and process evaluations within the category of implementation science. To be honest, I found this latter category of definitions rather unhelpful – they seemed to be so broad that almost anything could be labelled implementation science. So, I am going to choose to just go with the narrower understanding of the term.

Now I have to mention here that I thoroughly enjoyed the symposium and found implementation scientists to be a really fascinating group to talk with. And so, as a little gift back to them, and in recognition of the difficulties they are having in agreeing on a common definition, I have taken the liberty of creating a little pictorial definition of implementation science for them (below). I am sure they will be delighted with it and trust it will shortly become the new international standard ;-).
implementation science

So what else do you need to know about implementation science?

Well, it tends to be done in the health sector (although there are examples from other sectors) and it seems to focus on uptake by practitioners (i.e. health care providers) more than uptake by policy makers. In addition it is, almost by definition, quite ‘supply’-driven – i.e. it tends to focus on a particular evidence-informed intervention and then study how that can be implemented/scaled up. I am sure that this is often a very useful thing – however, I suspect that the dangers of supply-driven approaches that I have mentioned before will apply; in particular, there is a risk that the particular evidence-informed intervention chosen to be scaled up, may not represent the best overall use of funds in a given context. It is also worth noting that promoting and studying the uptake of one intervention may not have long-term impacts on how capable and motivated policy makers/practitioners are to take up and use research in general.

A key take home message for me was that implementation science is ALL about context. One of my favourite talks was given by @pierrembarker who described a study of the scale-up of HIV prevention care in South Africa. At first the study was designed as a cluster randomised controlled trial; however, as the study progressed, the researchers realised that, for successful implementation, they would need to vary the approach to scale-up depending on the local level conditions, and thus an RCT, which would require standardised procedures across study sites, would not be practical. Luckily, the researchers (and the funders) were smart enough to recognise that a change of plan was needed and the researchers came up with a new approach which enabled them to tailor the intervention to differing contexts, and at the same time generate evidence on outcomes which was as robust as feasible. Another great talk was given by Theresa Hoke of @FHI360 who described two programmes to scale up interventions that almost completely failed (paper about one of them here). The great thing about the implementation science studies were that they were able to demonstrate clearly that the scale-up had failed and to generate important clues for why this might be the case.

One final cool thing about implementation science is how multi-disciplinary it is; at the symposium I met clinicians, epidemiologists, qualitative social scientists and – perhaps most intriguingly – organisational psychologists. I was particularly interested in the latter because I think it would be really great if we could get some of these types involved in evaluating/investigating ‘demand-side’ evidence-informed policy work funded by organisations, including DFID, (the department formerly known as-) AusAID and AHSPR. These programmes are really all about driving organisational change, and it would be very useful to get an expert’s view on what approaches (if any!) can be taken by outside actors to catalyse and support this.

Anyway, sorry for such a long post but as you can tell I am really excited about my new discovery of implementation science! If you are too, I would strongly recommend checking out the (fully open access) Implementation Science Journal. I found the ‘most viewed’ articles a good place to start. You will also soon be able to check out the presentations from the symposium (including my talk in which I call for more unity between ‘evidence geeks’ like me and implementation scientists) here.


Capacity building rule 4: measure whether anything has been learnt

I suspect that one reason that bad capacity building programmes have persisted for so long is that monitoring and evaluation of capacity building has been so poor. It is commonplace for capacity building programmes to be ‘assessed’ almost entirely on the basis of subjective measurements of how much people have enjoyed the experience or how much they think they have learnt. Of course it is lovely that people enjoy themselves – but surely we should be trying a bit harder to find out if people have actually learnt anything.

There are some exceptions where more rigorous approaches have been used and they illustrate just how vital it is that we get a bit more objective in our assessments.

A multi-million pound science communication capacity building programme (which I won’t name!) had an independedent evaluation which compared outputs produced by participants before and after they took part in the scheme. The assessment found NO significant difference in the quality of outputs. A bit of a depressing finding.

A train the trainers workshop I ran used a diagnostic test before and after the course to test knowledge of basic principals of pedagogy. The test did reveal a significant increase in scores – although it was notable that a full third of participants continued to get the wrong answers even after the intensive course. But more worryingly, observations of teaching practices carried out in the months following the course revealed that many participants had reverted to their old, bad teaching habits. This certainly taught me of the importance of follow-up mentoring and within-workplace support for learning.

In both the above examples, participants themselves rated the capacity building programmes as excellent – further illustrating that people’s subjective view of the experience may differ significantly from a more objective assessment of what has been learnt.

I strongly believe that if we implemented better monitoring and evaluation of capacity building programmes, it would be quite depressing to start with because it would prove that lots of the stuff we are doing is not working. But it would provide a mighty big incentive for all of us to up our game and start adapting capacity building programmes so they could make a real difference.

So that’s it, those are my four simple rules. What do others think? Would you add other rules? Or do you think I am being to harsh on capacity building programmes, and they are actually generally better than I have implied? Thoughts welcomed!

Want to read the full series of 4 blogs? Start with this one here.