This morning I have been reading, with interest, Giel Ton's 2020 paper: Development policy and impact evaluation: Contribution analysis
for learning and accountability in private sector development
I have one immediate reaction; which I must
admit I have been storing up for some time. It is to do with what I
would call the hijacking of the meaning or definition of 'impact evaluation'. These days impact evaluation seems to be all
about causal attribution. But I think this is an overly
narrow definition and almost self-serving of the interests of those trying to promote methods specifically dealing with
causal attribution e.g., experimental studies, realist evaluation, contribution analysis and process tracing. (PS: This is not something I am accusing Giel of doing!)
I would like to see impact evaluations widen their
perspective in the following way:
1. Description: Spend time describing the many forms of impact a
particular intervention is having. I think the technical term here is multifinality. In a private-sector
development programme, multifinality is an extremely likely
phenomenon. I think Giel has in effect said so at the beginning of his
paper: " Generally, PSD programmes generate outcomes in a
wide range of private sector firms in the recipient country (and often also in
the donor country), directly or indirectly."
2. Valuation: Spend time seeking relevant participants’ valuations
of the different forms of impact they are experiencing and/or observing. I'm not talking here about narrow
economic definitions of value, but the wider moral perspective on how people value things - the interpretations and associated judgements they make. Participatory approaches to
development and evaluation in the 1990s gave a lot of attention to
people's valuation of their experiences, but this perspective seems to have
disappeared into the background in most discussions of impact evaluation these
days. In my view, how people value what is happening should be at the heart of
evaluation, not an afterthought. Perhaps we need to routinely highlight the stem of the
word Evaluation.
3. Explanation: Yes, do also seek explanations of
how different interventions worked and failed to work (aka causal attribution). Paying attention
of course to heterogeneity, both in the forms of equifinality and multifinality Please Note: I am not arguing that causal attribution should be ignored - just placed within a wider perspective! It is part of the picture, not the whole picture.
4. Prediction: And in the process don't be too dismissive of the
value of identifying reliable predictions that may be useful
in future programmes, even if the causal mechanisms are not known or
perhaps are not even there. When it
comes to future events there are some that we may be able to change or
influence, because we have accumulated useful explanatory knowledge. But there are also many which we acknowledge
are beyond our ability to change, but where with good predictive knowledge we
still may be able to respond to appropriately.
Two examples, one contemporary, one very old: If
someone could give me a predictive model of sharemarket price
movements that had even a modest 55% accuracy I would grab it and run,
even though the likelihood of finding any associated causal mechanism would probably
be very slim. Because I’m not a
billionaire investor, I have no expectation of being able to use an explanatory
model to actually change the way markets behave. But I do think I could respond in a timely way
if I had relevant predictive knowledge.
Similarly, with the movements of the sun, people
have had predictive knowledge about the movement of the sun for millennia, and this informed their agricultural practices. But even now that we have much improved
explanatory knowledge about the sun’s movement few feel this would think that this
will help us change the way the seasons progress.
I will now continue reading Giel's paper…
2021 02 19: I have just come across a special issue of the Evaluation journal of Australasia, on the subject of values. Here is the Editorial section.
