Over the past few months I have been engaged in discussions with other members of the Association of Professional Futurists (APF) Evaluation Task Force about how activities and outcomes in the field of foresight/alternative futures/scenario planning can usefully be evaluated.
Just recently the subject of Theories of Change has come up, and it struck me that there are at least three ways of looking at Theories of Change in this context:
The first perspective: A particular scenario (i.e. an elaborated view of the future) can contain within it a particular theory of change. One view of the future may imply that technological change will be the main driver of what happens. Another might emphasise the major long-term causal influence of demographic change.
The second perspective: Those organising a scenario planning exercise are also likely to have either explicitly or implicitly or mixture of both a Theory of Change of how their exercise is expected to influence on the participants, and the influence those participants will have on others.
The third perspective looks in the opposite direction and raises the possibility that in other settings a Theory of Change may contain a particular type of future scenario. I'm thinking here particularly of Theories of Change as used by organisations planning economic and/or social interventions in developed and developing economies. This territory has been explored recently in a paper by Derbyshire (2019), titled "Use of scenario planning as a theory-driven evaluation tool. FUTURES & FORESIGHT SCIENCE, 1(1), 1–13. In that paper he puts forward a good argument for the use of scenario planning methods as a way of developing improved Theories of Change. Improved in a number of ways. Firstly a much more detailed articulation of the causal processes involved. Secondly, more adequate attention to risks and unintended consequences. Thirdly, more adequate involvement of stakeholders in these two processes.
Both the task force discussions and my revisiting of the paper by Derbyshire have prompted me to think about the potential use of a ParEvo exercise as a means of articulating the contents of a Theory of Change for a development intervention. And to start to reach out to people who might be interested in testing such possibilities. The following possibilities come to mind:
1. A ParEvo exercise could be set up to explore what happens when X project is set up in Y circumstances with Z resources and expectations. A description of this initial setting would form the seed paragraph(s) of the ParEvo exercise. The subsequent iterations would describe the various possible developments that took place over a series of calendar periods, reflecting the expected lifespan of the intervention, and perhaps a limited period thereafter. The participants would be, or act in the role of, different stakeholders in the intervention. Commentators of the emerging storylines could be independent parties with different forms of expertise relevant to the intervention and its context.
2. As with all previous ParEvo exercises to date, after the final iteration there would be an evaluation stage, completed by at least the participants and the commentators, but possibly also by others in observer roles. You can see a copy of a recent evaluation survey form here, to see the types of evaluative judgements that would be sought from those involved and observing.
3. .3. There seemed to be at least two possible ways of using the storylines that have been generated, to inform the design of a Theory of Change. One is to take whole storylines as units of analysis. For example, a storyline evaluated as both most likely and most desirable, by more participants than any other storyline, would seem an immediately useful source of detailed information about a causal pathway that should go into a Theory of Change. Other storylines identified as most likely but least desirable would warrant attention as risks that also need to be built into a Theory Of Change, along with any potential means of preventing and/or mitigating those risks. Other storylines identified as least likely but most desirable would warrant attention as opportunities, also to be built into a Theory Of Change, along with means of enabling and exploiting those opportunities.
4. 34. The second possible approach would give less respect to the existing branch structure, and focus more on the contents of individual contributions i.e. paragraphs in the storylines. Individual contributions could be sorted into categories familiar to those developing Theories of Change: activities, outputs, outputs, and impacts. These could then be recombined into one or more causal pathways that the participants thought was both possible and desirable. In effect, a kind of linear jigsaw puzzle. If the four categories of event types were seen as being too rigid a schema (a reasonable complaint!), but still an unfortunate necessity, they could be introduced after the recombination process, rather than before. Either way, it probably would be useful to include another evaluation stage, making a comparative evaluation of the different combinations of contributions that had been created. Using the same metrics as are already being used with existing ParEvo exercise.
More ideas will follow..
The beginnings of a bibliography...
3
3
I see a lot of value in what you’re suggesting. Too often I’ve seen Theories of Change used to support an accountability and audit function. They are more Strategies of Change, with the four components: activities, outputs, outputs, and impacts used initially to determine program budget and subsequently used to justify return on investment.
ReplyDeleteThe perspective that a Theory of Change is a hypothesis that must be tested is more focused on learning. With such a perspective, the design of the theory would be more precise if it “anticipated” as many scenarios as practically possible, both positive and negative. Developmental evaluations do this on the fly, but presuppose the initial design is accurate. Hence, they can experience significant learning, largely referenced to design imperfections.
The ParEvo exercise used in the first way you suggested – with the storylines used in their entirety, is more appealing to me than thematically categorizing the various paragraphs. I suspect that stems from a belief that reducing a storyline to its individual component parts, particularly in a complex system, feels as if it must be losing an understanding of the interplay between the complex forces that the Theory of Change is attempting to address. That said, in an immature field, one in which there is an absence of literature, the second “jigsaw” approach would still help program designer consider a broader range of factors and make better decisions on the type of intervention and activities to include and possibly avoid.