Sometimes I have the feeling that development experts want to paint the debate about randomised controlled trials as more polarised than it really is. They seem to think they are fighting against a maniacal, narrow-minded, statistics-obsessed pro-RCT bogeyman who they fear is about to take over all development funding. It may seem that they are committing the classic logical fallacy of painting the opposing argument as ludicrous in order to strengthen their point – but having had many discussions on this topic I believe that many people really do believe in this bogeyman. So, here is my attempt to kill him!
I think RCTs can give some important information on some questions which can help us to make some decisions in the international development sector. BUT this does not mean that I think that RCTs are the only form of evidence worth considering or that they are the best method in all cases – and I am not sure that anyone does think this.
I get frustrated that every time I mention something about RCTs people seem to respond by arguing with things that I have not actually said! For example I often get people telling me that RCTs don’t necessarily tell us about individual responses (I agree), that many interventions cannot practically be evaluated by RCTs (I agree), that the results of RCTs may not be transferable to different contexts (I agree), that policy decisions are made on other factors than just evidence of ‘what works’ (I agree), that scientists often get things wrong (I agree -in fact I even have a blog post on it!) etc etc
What bothers me is that by focussing on a debate that doesn’t in truth exist, we are missing the opportunity to have much more useful and interesting discussions. For example, I would love it if more people thought about more innovative ways of integrating RCTs (or quasi-experimental approaches) with qualitative research so that both arms add value to the other. A valiant attempt is described here but as you can see, the poor social scientists were frustrated by the ‘hard’ scientists’ unwillingness to change their protocol. I wonder if qualitative research could be integrated more easily into adaptive trial designs which are designed so that protocol can be changed as research is gathered?
Another discussion that I would love to hear more on is how do we measure impacts such as behaviour/attitudes/capacity in a more objective way (a discussion which is validwhether one is using a RCT approach or not)? I feel that too much evaluation in international development relies on self-reporting of these things – for example measuring whether people report an increase in ca
pacity following a training event. When I worked at INASP we did some work to compare self-reported ability (of policy makers) to use research with actual ability as measured using a diagnostic test (sorry – this work is still ongoing but I hope it will be published eventually). Similarly, an excellent piece of work here looked at researchers’ reported barriers to using journals with their actual skills in using them while this report compared the work of the Ugandan parliament in relation to science and technology with the reported competence of MPs. In all cases there was little correlation. This makes me think that we need to be more creative in measuring actual impacts -rather than relying so much on self-reporting.
These are just a couple of the interesting questions that I think we could be exploring – but I am sure if people could stop spending their time fighting the RCT bogeyman, they could come up with a lot more interesting, valid and important questions to discuss.