In a previous post I talked about the issue of many capacity building initiatives using self-reported ability as a measure of impact. To further illustrate this point, I decided to carry out a small scientific experiment. I gathered a randomly selected group of study participants and gave them the following instructions:
1. Rate their tiddly wink playing ability on a scale of 1 (dreadful) to 5 (outstanding)
2. Tiddle their wink (no winking of tiddles was allowed)
3. Measure how far they had tiddled their wink (in cms)
(I suspect that the above may be the best sentence I will EVER blog)
The results are shown in the graph below. Now as any scientist will tell you, the R^2 value written on this graph indicates that this (poorly designed, probably manipulated ;-)) experiment definitively proves that my pre-existing prejudices were correct i.e. self-reported ability to play tiddly winks is not correlated with actual ability.
So what is the conclusion of this? Well, for one thing, if you ever see a log frame for a project to build tiddly winking capacity with ‘increase in self-reported tiddly wink ability’ as the verifiable indicator, I trust you will refer the project team to this definitive rebuttal.
But in addition, I think we should be a lot more sceptical of any project which uses perceived increase in ability as the measure of success. Of course, there may be some abilities which ARE correlated with self-reported ability. But I suggest that this correlation needs to be demonstrated before self-reporting can be used as a reliable proxy indicator.