Schooling does not equal learning

School in East Timor Worldbank photostream

One of the major aims – and indeed major successes – of the millennium development goals, has been to increase the number of kids going to school. At first glance, it appears wonderful that enrollment in schools went from 50% to 66% between 1995 and 2010. But the worrying thing is, that getting more kids into schools does not necessarily mean that they are learning more. In fact, a recent report from the Centre for Global Development reveals that the levels of educational attainment amongst children in developing countries are worrying low. The report draws on large global datasets but a few examples which stood out for me include:

• In India, 60% of grade 8 children are unable to use a ruler to measure a pencil while only 27% who finish primary school can carry out tasks (such as reading a passage of text and telling the time) that are expected to be achieved by the end of the second year of school.
• In Tanzania and Uganda, less than half of children aged between 10 and 16 have basic literacy and numeracy skills.
• In Malawi, almost 80% of sixth graders score below the international minimum standard for reading proficiency
• An average eighth grader in Ghana achieves a test score in maths and science which is equivalent to the lowest 0.2% of US students

I have worked in international development for many years but I still find these figures truly shocking. I can’t help wondering if a huge amount of the work we do in capacity building at organisational and institutional levels might be unnecessary if only people were getting a decent standard of education from the outset. I am reminded that at the International Conference on Evidence-Informed Policy Making in Nigeria last year, one of the main conclusions was that if policy makers are ever going to be able to make use of research evidence, they need to have much better levels of basic education.

So how can we improve learning? The report talks a lot about the use of assessment to improve educational outputs – not so much because assessment drives learning but because poor assessment results drive people to reform the system. At present it seems that many people in developing countries are not aware of how poor the education system is and are therefore not demanding reform (see for example the graph on the left – taken from the report). But precisely what type of reform would work is less clear.

There is evidence that teacher incentives can improve both attendance and effort – for example this systematic review suggests that having teachers on fixed term rather than permanent contracts increases attainment. However the results are patchy and a recent study from Kenya showed that contract teachers only improved attainment when hired they were hired by NGOs rather than the government. Of course, getting the teachers to actually turn up is important (!) but getting teachers to promote a friendly learning environment (e.g. encouraging kids to ask questions) is equally crucial and equally challenging. A number of studies (see for example here) have shown that changing the culture of teaching to a more learner-centred approach is very difficult. I have experience of this myself – I used to teach learner-centred pedagogy to capacity building trainers. The work was great fun, but hard; people’s experience of learning is very personal and deep-seated and it can be quite scary for people to break free from this.

Overall, my conclusion from reading this report is that we know shockingly little about how we can improve education. We have failed to invest in good quality research about how we can support people learn and academic pedagogy has been dominated by pseudoscience. Education research has lagged behind other areas of development research for too long and getting our kids to actually learn is just too important to neglect.

Related articles

• Schooling Is Not Education: Using Assessment to Change the Politics of Non-Learning (jiandae.com)

Community Driven Development – would it work in Vauxhall?

Community Driven Development (CDD) is an approach to development, often used in fragile or conflict affected areas, which aims to engage the community in the decision making process to decide how development funds should be spent in the local area. It typically involves setting up some sort of community council which then makes decisions on how a block grant should be invested. It is massively popular with development agencies – the World Bank, for example, supports over 400 CDD projects in 94 countries at a cost of over $30 billion. Its supporters are almost evangelical about its ability to rebuild damaged societies and promote social cohesion but does it really work? That is the question asked in a recent critical review of CDD in conflict-affected areas written by Elisabeth King. And the answer is… not really.

The review is a masterpiece of clear-headed analysis and intelligently combined research methodologies. King began by constructing a general Theory of Change based on the programme documents of previous interventions to describe how people believe that CDD will lead to social cohesion. She then carried out a synthesis of rigorous impact evaluations of CDD programmes in conflict-affected areas to test whether the included studies demonstrated an effect on the intermediate and final outcomes identified in the Theory of Change. And finally she carried out qualitative research to understand the views of those implementing and researching CDD.

The review demonstrates that CDD programmes can be an effective way to distribute development funds to achieve ‘proximate outcomes’ – for example, roads got built and people got grants. But there is little evidence that it achieves the kind of long-term social outcomes that its implementers are setting out to achieve. None of the five included studies found evidence of improved governance and only one out of the four which looked at it found any evidence of an improvement in social cohesion. The qualitative arm of the study revealed that some experts felt that the intervention was being over-hyped – they described the generic Theory of Change on which CDD interventions are based as “lofty”, “unrealistic”, “inherently flawed” and even “ridiculous”.

Would CDD make Vauxhall a happier and more cohesive place?

I began to understand this scepticism when I considered what would happen if it was implemented in my own area. I live in the London area of Vauxhall. Like most areas of London, it is fantastically diverse. It is well-known for its large gay community – most of my neighbours are gay couples – but perhaps surprisingly, it also has big populations of quite conservative Christians. If you get on a bus you are as likely to hear Portuguese or Yoruba as English – and a massive 40% of people living here were not born in the UK. Around a third of the accomodation is social housing including some big housing estates and high-rise flats – but we also have some very gentrified streets. I would not say that we have massive problems with intergroup intolerance – but neither would I say that it is a cohesive community; in fact, there is relatively little interaction between the demographic groups. So, would CDD bring us all together?

Well… I’m not convinced. If an NGO came along and said that our area could have a million pounds to spend on something so long as we all got together to agree on it, I can’t really see how that would bring me closer to those around me. First, I would have to decide to go to the meeting – and although a million pounds is a lot of money, I am not certain that I would feel that it would be likely to have a sufficiently big impact on me that I would give up my precious free time to attend. And if I did go, and if I found that there were others who had massively different ideas from me on how it should be spent, I suspect it might make me feel less connected to them rather than more.

So what next for CDD? King is careful not to suggest that the approach should be discarded completely. As discussed above, it can be an effective way to achieve proximal outcomes (e.g. infrastructure, service delivery) and, given the relative lack of tools we have for working in fragile and conflict-affected environments, this is not to be sniffed at. However, she is also clear that we cannot continue to treat CDD as a panacea. She concludes that we need to be guided by “humility and more realistic goals”; a suggestion that we would all do well to consider.

This review, along with a number of previous articles, suggests that we need to think seriously about CDD – can the approach be modified to improve its outcomes? Or should it be seen as just a mechanism to deliver funding for projects but not as a means to social cohesion? It will be interesting to see how those implementing CDD respond.

Evidence synthesis – what has it ever done for us?

I have talked before about the danger of using results from single research studies to push for policy change. A more balanced view of the whole body of evidence can be gained by carrying out evidence synthesis.  Systematic reviews (or other rigorous synthesis approaches) attempt to gather, appraise and summarise bodies of evidence in a transparent way. By looking at a whole body of evidence, and appraising the rigour of the studies you are looking at, you can get more certainty about what is really going on.

Systematic reviews have long been used in the medical field and have been shown to provide more accurate results than relying on clinical ‘expertise’. In (non-medical) international development topics, rigorous synthesis is much less established; there are relatively few people with expertise in synthesis and the methodologies for synthesising social science research results, and in particular qualitative data, are still being developed.

Nevertheless, synthesised evidence is starting to reveal new and important information about international development topics. Here I summarise three important roles that synthesised evidence can play in improving development interventions.

1. It can tell us that something is true, which we didn’t realise was true

*OK so maybe everyone else knew about the existence of narwhals but it came as a bit of a surprise to me when I discovered them in a David Attenborough documentary last year…

Evidence synthesis can sometimes reveal something to be true which an ‘unweighted’ or non-systematic view of the literature would not have revealed. A good example is this paper about decentralisation of services in developing countries. The authors conclude the following:“Many influential surveys have found that the empirical evidence of decentralization’s effects on service delivery is weak, incomplete and often contradictory. Our own unweighted reading of the literature concurs. But when we organize the evidence first by substantive theme, and then – crucially – by empirical quality and the credibility of its identification strategy, clear patterns emerge. Higher quality evidence indicates that decentralization increases technical efficiency across a variety of public services, from student test scores to infant mortality rates.” In other words, only by taking the evidence together and organising it by quality were the authors able to reveal the real role that decentralisation is playing.

Back in the 80s, we thought we knew it all…

2. It can tell us that something that we all thought was true is actually not true

A classic example of this was this systematic review published last year which showed that, contrary to popular belief in the development community, routine deworming has little impact on school attendance or school performance. Unsurprisingly, this finding was pretty controversial – development ‘experts’ had been waxing lyrical about deworming as a means for educational improvement for years. However, by looking at the evidence together and, crucially, looking at the quality of the evidence, the authors revealed a different story; much as we liked the idea of being able to improve educational outcomes with an inexpensive pill, the evidence revealed, it’s just not that simple.

This is me in high school – looking smug and not suspecting that quantum physics would make my head explode.

3. It can tell us that something we thought we fully understood, we actually don’t have a clue about

The Justice and Security Research Consortium recently carried out a synthesis of the evidence on the media and conflict (it is not quite published yet but a summary can be found here). They found a lot of papers which make big claims about the media’s role either in promoting and preventing conflict. The large body of literature making these claims could easily fool a busy policy maker into assuming that the links between the media and conflict were well-established. However, when the evidence was assessed for rigour, it was found that many of these papers were based only on opinion or theory and that the number of high quality research papers in this area was low. They summarised that, at present, it is not possible to confirm or refute the claims about the media’s role in conflict based on the available evidence. Now some might say that this is the problem with synthesis – it often just tells us that we don’t know anything much! But in fact for a policy maker it is very important to know whether an intervention is tried and tested in multiple contexts or whether it is an innovative strategy which may have impact but which it would be sensible to monitor closely.

So, synthesised evidence – it might not sound exciting, but it is actually revealing lots of exciting new things. To find out more about what synthesised evidence can tell us check out this database of international development systematic reviews. And watch out for a follow-up post on how synthesised evidence can be communicated effectively.

Science and development – getting beyond the hyperbole…

I have a fairly short attention span so when I attend long international development meetings, I like to indulge in a little bit of conference bingo. For the un-initiated, this involves writing down a list of your favourite development clichés, and then giving yourself a point (and a healthy dose of smugness) every time someone mentions one of them. I like to include any mention of silos or capacity building, any statement about the GDP of (insert Asian country) once having been the same as the GDP of (insert African country), a statement about (insert developing country) not being poor but just being mismanaged, and, my favourite, a statement suggesting that science and technology are critical to development.

Given that I like science and I like development, you might expect me to like statements linking the two – but the reason it annoys me, is that when you then try to follow up and ask how science is going to lead to development, people often either get quite vague or give answers which are not really based on reality. It sometimes seems that the link between science and development is some kind of mantra which is backed up mainly by quoting other important people who have put it forward. I actually DO think that science can feed into development but I suspect that does not always happen in the ways people expect. I therefore believe that it is worth unpicking a little bit the causal pathways by which science can contribute to development.

So, let’s examine a few of the ways in which science might contribute to development.

1. Investment in science leads to economic growth

This is a commonly held assumption and is often cited by politicians as a reason for investing in science. However, the evidence on whether investment in scientific research is a valuable way to stimulate the economy in developing countries is weak. One justification that I have heard many times is that countries such as Korea, which invested in science and technology subsequently saw economic growth. However, it is clear that there were many factors that led to Korea’s economic growth including major policy changes related to trade and thus there is no particular reason to assume that investment in science made all (or even any of) the difference. There are some studies from richer countries which have attempted to calculate the economic payback of investment in research, but there is little from developing countries. Furthermore, there is clearly not a binary decision between investment in research or not. Investment in poor research, as unfortunately is common in countries without robust processes for selecting high quality research, is very unlikely to lead to any economic growth. My suspicion is that the link between economic growth and investing in scientific research in most developing countries is a testable hypothesis, not a proven fact but I would be grateful if you know more about this topic and you would like to point me in the direction of some good relevant research.

2. Investment in research provides new tools and products which will benefit the poor

It is very seductive to think that scientific research will lead to new technologies which will save the world. Indeed there are some great examples of new products which have had a significant impact on developing countries – anti-retroviral treatment, polio vaccines and scuba rice are just a few amazing inventions. However, I think a degree of caution is required given that there are lots of existing ‘solutions’ to problems that are not currently being used. For example, we know how to treat many neglected tropical diseases, we know how to purify water and we know how to build roads that resist flooding – but we also know that none of these solutions are being universally adopted. For this reason, I do believe that finding new solutions to problems is important, but I think we must not forget that an equally big challenge is making sure that the solutions we have found are actually used.

3. Research will inform policy making

This is an area in which I feel that science – and scientific approaches – can really make a difference. However, I would still add some important caveats. The first is that research information will only help policy makers to make better decisions if they are in fact motivated to make good decisions in the first place! If a policy maker is only interested in improving her own lot in life, they are unlikely to be motivated to use research information to improve life for the general public. Secondly, even if policy makers are motivated to improve conditions, they need to have a certain degree of evidence literacy to understand and interpret the implications of research evidence. Therefore, I think that efforts to promote evidence-informed policy need to consider the ‘demand-side’  – both the motivations of policy makers and their abilities to understand and use research.

4. Spillover effects

This is perhaps the least discussed way in which science can impact on development, but I sometimes wonder if it might be the most important. What I mean by spillover effect is the general increase in scientific thinking that you get if you invest in research. This happens because some scientists will leave to enter new professions (journalism, politics etc.) and also because when you invest in research within academic institutions, you get a more research-literate faculty who in turn provide better training in scientific thinking to the students they are teaching. My hypothesis (and given the paucity of evidence, it is just a hypothesis albeit one which I suspect is true ) is that the resultant critical thinking, questioning and problem solving skills could be a key factor in developing the human capital needed to run effective institutions and a populace more able to scrutinise policy makers.

I’m actually writing this blog in preparation for a talk I am giving next week so, do please do comment, enlighten me or even (politely) disagree – hopefully your input will improve my talk and thereby reduce the number of people playing conference bingo during it!

All photos courtesy of http://www.morguefile.com

The enthalpy of aid

I have decided to continue my theme of exceedingly geeky analogies by today comparing international development to a chemical reaction. I don’t know if you remember when you learnt chemistry at school but one of the few things I can remember is drawing these little ‘enthalpy graphs’ that show how reactions proceed. The graph shown here could for example represent setting fire to a very flamable thing – you need to put a bit of energy in at the beginning, but then the overall you get more energy out than you put in. The little bit of energy you put in at the beginning is called the activation energy.

I think that the very best international development projects are similar to this activation energy. They are projects which inject a little bit of additional funds, know-how, coordination or whatever to people and organisations who already have huge potential to make a difference. This is why I really like the new advertising campaign from Oxfam America - it highlights how relatively small amounts of investment can be enough to support people to really make a difference. It makes a welcome change to the dominant narrative in international development fundraising campaigns – think b-list celebrity rowing down a remote African river, giving out bed-nets to the poor (but happy) Africans. The Oxfam campaign, in contrast, highlights people living and working in developing countries who are working day in day out to make a difference in their community – and it demonstrates that the contribution that external actors can play can be important but is relatively minor.

I think many capacity building programmes would do well to consider the ‘activation energy’ phenomena too. The fact is that capacity buildng programmes – whether they involve mentoring, training, organisational development or whatever – generally only make a marginal change. As an example, lets say that capacity building will lead to a 5% increase in ‘capacity’ to do something (yes I know you can’t measure capacity as a percentage but bear with me…!). You can see that if you offer that programme to people who have 30% existing capacity, it will increase their capacity but they still won’t really be able to do much. But if you are able to work with people who already have 95% capacity, your programme could be just the little bit extra input that is needed to make a real difference.

There are some ethical concerns about this approach – some would say that it is unfair to offer support to those who are already doing relatively well. I think this is a fair comment but I also think that we need to be really honest about how much a given project can ever achieve. If it can only ever offer relatively minor increases in effectiveness, our only choice is to provide this to those who could make use of that to make a real difference. If we aim to support those who are far less able to improve their own situation and those of others then we may need to rethink our project.

Of course this approach completely relies on being able to identify those who have the existing potential – the people and organisations who are at 95%. And unsurprisingly, this is easier said and done! But maybe that is another blog post…

Adapt!

Over the holidays I read Tim Harford‘s book ‘Adapt‘. In it, he discusses how various tricky problems (from running a sucessful business to solving world poverty) are best tackled using an evolutionary approach. He discusses three steps to sucessful ‘evolution’: first you need a variety of possible solutions, you need to make sure that if any solution fails (and many will) you can survive it and finally you need to identify which of the many solutions works best in the context you are dealing with.

I sometimes find books which attempt to explain all sorts of things using a central model or metaphor a bit annoying and contrived – but Adapt really rang true for me. When I was pondering this, I realised that I liked it so much because I already have a tendency to view life throught the lens of evolution. I used to research immunology and the processes that go on in our bodies as we fight diseases are remarkably good metaphors for understanding the world. By looking at the immune system you can see the results of ‘classical’ evolution (i.e. the selection of sucessful genes over long time periods). But the immune system also has a special adaptive arm that displays an accelerated form of adaptation – kind of like evolution on speed.

So, my dear readers, I thought that I would go ahead and give you a wee immunology lesson (you’re welcome) because I think it can help us all to understand the world!

In your blood there are lots of cells including some special ones which form the adaptive immune system. These cells are called T-cells and B-cells and they are like adaptation Jedi masters. There are a few different sub-types of both T and B cells but essentially they all work in a similar way. To start with, your body produces a whole load of different cells with sticky ‘receptors’ on them. Each cell has only one kind of receptor but the receptors on different cells are slightly different.

Now, when a germ comes along, because there are so many different immune cells with so many different sticky receptors on them, it will eventually stick to one of them. Even better, the cells have a built-in feedback system so they recognise ‘danger signals’ which are like little red flags that indicate that whatever the cell has stuck to is a ‘baddie’. When a B-cell or T-cell sticks to a dangerous germ, the feedback mechanism kicks in, and very quickly that one lucky cell multiplies into a immune system army – each member of which is specific for the particular germ that the first one encountered.

And the army of cells sets off to kill the germs in a variety of interesting ways.

Once all the germs have been killed, the immune system more or less goes back to normal but with one key difference – a small sleeper ‘cell’ (no pun intended) is maintained with specificity for that particular germ. It means that if you encounter the same germ again, the body can get rid of it much faster. This explains why vaccines work – the vaccine promotes an inital reaction against a germ (or a part of a germ) so that when you encounter the actual germ you have the ability to get rid of it quickly.

I think the adaptation of immune cells is a good metaphor for many of the stories that Tim Harford describes in ‘Adapt’. Whether you are running a business or trying to solve world poverty, it is a good idea to try out a whole variety of solutions. In fact, if you want to be like the immune system, you might want to be designing and trying out new ways of doing things before you even think you have a problem! You also need to make sure that when one of your solutions ‘works’, you have a good feedback system in place so that this is recognised and that there is capacity to rapidly scale it up. And crucially, if the particular ‘problem’ disappears, you want to make sure that you don’t forget about the solution you came up with. You should make sure that the capacity to deal with a similar problem in the future is maintained.

So, what did I learn from ‘Adapt’? Although it focussed on some rather ‘macro’ issues, it actually made me think a lot about organisations and the qualities of a good leader or manager. The book makes it clear that taking an evolutionary approaches to problem solving is not an alternative to good leadership. In fact, enabling an organisation (whatever it may be) to adapt and deal with problems in this evolutionary manner could be seen as the hallmark of trully great leaders. Without such leadership, the natural tendency of people to innovate and experiment becomes crushed by mindless adherance to rules and lack of delegated authority. This point is illustrated brilliantly in the book by examining different approaches to leading the US military in recent years but I am sure we can all also think of organisations where we have encountered the same problem on a smaller scale.

Luckily, Harford is not the only person to be thinking along these lines. I was lucky to have the opportunity to do a diploma in management a couple of years ago and I discovered that there is a huge body of  work on organisational development examining how organisations can become more adaptive and healthy. Essentially, much of this work is promoting a similar approach to that described by Harford: enable local-level innovation, make sure you can recognise success and then have systems which allow successful solutions to be scaled up efficiently.

In conclusion, I highly recommend reading Adapt. And, I liked it so much that I have nominated it as book of the month on this business book club that I am a member of – its free to join so feel free to pop in to join in the discussion.

Guest post for Africa on the Blog

This is my cat sitting in a basket. The basket is from Malawi, which is of course in Africa… so I figured its sort of tangentially linked to the theme of this post…

I was really delighted a couple of weeks ago to be invited by Ida Horner, editor of the wonderful Africa on the Blog, to contribute a guest post. My only instructions were to write something related to Africa and so I decided to take the opportunity to do a little bit of research into the perceptions towards Africa amongst my European friends and family. The finished article is here – hope you like it.But don’t just look at my piece – I strongly recommend you stop a while and check out some of the great writing on that site. For a start I would recommend their end of 2012 list of African gold medalists – a really inspiring and surprising list. And for balance you may wish to check out the rather less uplifting sin bin list.

A few other of my favourites from the last year are:

Why does adversity fail to reveal African genius by David Mpanga

Homosexuality in Malawi – how objective is the local media? by Jimmy Kainja

Africa rising into the mainstream – is this what we want? by Ossob Mohamud

What American’s can learn from Africans by ???

Enjoy!