Finding useful distinctions between different futures

 

This blog posting is a response to Joseph Voros's informative blog posting about the Futures Cone. It is a useful contribution in as much as it helps us think about the future in terms of different sets of possibilities. Here is a copy of his edited version.

Figure 1: Voros, 2017

My alternative, shown below, was developed in the context of supporting ParEvo.org explorations of alternative futures. It has some similarities and differences. For a start, here is the diagram.

Figure 2: Sets and sub-sets of alternative futures Davies, 2023

I will now list Joseph's explanation of each of the terms he used, and how they might relate to mine (in red)

• Possible – these are those futures that we think ‘might’ happen, based on some future knowledge we do not yet possess, but which we might possess someday (e.g., warp drive). I think these fall in the grey area above (which also contain the dark and light green).
• Plausible – those we think ‘could’ happen based on our current understanding of how the world works (physical laws, social processes, etc).I think these fall somewhere within the green matrix
• Probable – those we think are ‘likely to’ happen, usually based on (in many cases, quantitative) current trends. These probably fall within the Likely row of the green matrix
• Preferable – those we think ‘should’ or ‘ought to’ happen: normative value judgements as opposed to the mostly cognitive, above. There is also of course the associated converse class—the un-preferred futures—a ‘shadow’ form of anti-normative futures that we think should not happen nor ever be allowed to happen (e.g., global climate change scenarios comes to mind).These probably fall within the Desirable column of the green matrix
• Projected – the (singular) default, business as usual, ‘baseline’, extrapolated ‘continuation of the past through the present’ future. This single future could also be considered as being ‘the most probable’ of the Probable futures. As suggested above, probably at the most likely end of the Likely row in the above green matrix
• (Predicted) – the future that someone claims ‘will’ happen. I briefly toyed with using this category for a few years quite some time ago now, but I ended up not using it anymore because it tends to cloud the openness to possibilities (or, more usefully, the ‘preposter-abilities’!) that using the full Futures Cone is intended to engender. Probably also at the most likely end of the Likely row in the above green matrix

Preposterious events are not really covered. Perhaps they are at the extreme end of the Unlikely events with known probabilities i.e zero likelihood.

 

Though lacking in alliteration my schema does have some more practically useful features

The primary additional feature is that for each different kind of future there are some conjectured consequences in terms of likely appropriate responses. Some of these are shown red:

• Organisational "slack" i.e. uncommitted resources or reserves that could enable responses to the unforeseen (though, of course,  not every kind of unforseen event)
• Fringe investments, such as blue sky research, can be appropriate where a possibility is in sight but its likelihood of happening is far from clear
• Robust responses are those that might work, though not necessarily be the most effective or most efficient, across a span of possibilities having varying probabilities and desirabilities
• Customised responses are those more tailored to specific combinations of un/likely and un/desirable events. The following more detailed version of the green martix describes some major possible variations of this kind

Figure 3

Where to next?

I would like to hear from readers their views on the possible utility of these distinctions. And whether any other distinctions could be added to or replace those I have used. 

Postscript 2025 01 22

I came across this useful matrix view in
Luís, A., Garnett, K., Pollard, S. J. T., Lickorish, F., Jude, S., & Leinster, P. (2021). Fusing strategic risk and futures methods to inform long-term strategic planning: Case of water utilities. Environment Systems & Decisions, 41(4), 523–540. https://doi.org/10.1007/s10669-021-09815-1

How can evaluators practically think about multiple Theories of Change in a particular context?

This blog posting is been prompted by participation in two recent events. One was some work I was doing with the ICRC, reviewing Terms of Reference for an evaluation.  The other was listening in as a participant to this week's European Investment Bank conference titled "Picking up the pace: Evaluation in a rapidly changing world". 

When I was reviewing some Terms of Reference for an evaluation I noticed a gap which I have seen many times before. While there was a reasonable discussion of the types of information that would need to be gathered there was a conspicuous absence of any discussion of how that data would be analysed. My feedback included the suggestion that the Terms of Reference needed to ask the evaluation team for a description of the analytical framework they would use to analyse the data they were collecting.

The first two sessions of this week's EIB conference were on the subject of foresight and evaluation. In other words how evaluators can think more creatively and usefully about  possible futures – a subject of considerable interest to me. You might notice that I've referred to futures rather than the future, intentionally emphasising the fact that there may be many different kinds of futures, and with some exceptions (e.g. climate change) is not easy to identify which of these will actually eventuate.

To be honest, I wasn't too impressed with the ideas that came up in this morning's discussion about how evaluators could pay more attention to the plurality of possible futures. On the other hand, I did feel some sympathy for the panel members who were put on the spot to answer some quite difficult questions on this topic.

Benefiting from the luxury of more time to think about this topic, I would like to make a suggestion that might be practically usable by evaluators, and worth considering by commissioners of evaluations. The suggestion is how an evaluation team could realistically give attention not just to a single "official"  Theory Of Change about an intervention, but to multiple relevant Theories Of Change about an intervention and its expected outcomes. In doing so I hope to address both issues I have raised above: (a) the need for an evaluation team to have a conceptual framework structuring how it will analyse the data it collects, and (b) the need to think about more than one possible future and how that might be realised i.e. more than one Theory of Change.

The core idea is to make use of something which I have discussed many times previously in this blog, known as the Confusion Matrix – to those involved in machine learning, and more generally described simply as a truth table - one that describes four types of possibilities. It takes the following form:

In the field of machine learning the main interest in the Confusion Matrix is the associated performance measures that can be generated, and used to analyse and assess the performance of different predictive models.  While these are of interest, what I want to talk about here is how we can use the same framework to think about different types of theories, as distinct from different types of observed results.

There are four different types of Theories of Change that can be seen in the Confusion Matrix. The first (1) describes what is happening when intervention is present and the expected outcome of that intervention is present. This is the familiar territory of the kind of Theories of Change that an evaluator will be asked to examine.

The second (2) describes what is happening when intervention is present and the expected outcome of that intervention is absent. This theory would describe what additional conditions are present, or what expected conditions are absent, which will make a difference – leading to the expected outcome being absent.  When it comes to analysing data on what actually happened identifying these conditions can lead to modification of the first (1) Theory of Change such that it becomes a better predictor of the outcome and there are fewer False Positives (found in cell 2). Ideally the less False Positives the better. But from a theory development point of view there should always be some situations described in cell 2 because there will never be an all-encompassing theory that works everywhere. There will always be boundary conditions beyond which the theory is not expected to work. So an important part of an evaluation is not just to refine the theory about what works (1) but also to refine the theory of the circumstances in which it will not be expected to work  (2),  sometimes known as conditions or boundary conditions.

The third theory (3) describes what is happening when the intervention is absent but nevertheless the outcome is present. Consideration of this possibility involves recognition of what is known as "multi-finality" i.e. that some events can arise from multiple alternative causal conditions (or combinations of  causal conditions).  It's not uncommon to find advice to evaluators that they should consider alternative theories to those they are currently focused on. For example in the literature on contribution analysis. But it strikes me that this is often close to a ritualistic requirement, or at least treated that way in practice. In this perspective alternative theories are a potential threat to the theory being focused on (1). But a much more useful perspective would be to treat these alternative theories as potentially useful other courses of an action that an agent could take, which warrant serious attention in their own right. And if they are shown to have some validity this does not by definition mean that the main theory of change (1) is wrong. It' simply means that there are alternative ways of achieving the outcome, which can only be a bonus finding. 

The fourth theory describes what is happening when intervention is absent and the outcome is also absent (4).  In its simplest interpretation, it may be that the actual absence of the attributes of the intervention is the reason why the outcome is not present. But this can't be assumed. There may be other factors which have been more important causes. For example the presence of an earthquake, or the holding of a very contested election. This possibility is captured by the term "asymmetric causality" i.e. that the causes of something not happening may not simply be the absence of the causes of something happening. Knowing about these other possible causes of desired outcome not happening is surely important, in addition to and alongside knowing about how an intervention does cause the outcome. Knowing more about these causes might help other parties with other interventions in mind move cases with this experience from being True Negatives (4) to being False Negatives (3)

In summary, I think there is an argument for evaluators not being too myopic when they are thinking about Theories of Change they need to pay attention to.  It should not be all about testing the first (1) type of Theory of Change, and considering all the other possibility is simply as challengers, which may or may not then be dismissed  Each of those other types of theories (2-3-4) are important and useful in their own right and deserve attention.

Four types of futures that should be covered by a Theories of Change

ParEvo.org is a web app that enables the collaborative exploration of alternative futures, online. In the evaluation stage, participants are asked to identify which of the surviving storylines fall into each of these categories:

• Most desirable
• Least desirable
• Most likely
• Least likely

In one part of the analysis of storylines generated during a ParEvo exercise the storylines are plotted on scatter plot, where the two dimensions are likelihood and desirability, as seen in this example

Most Theories of Change that I have come across, when working as an evaluator, focus on a future that is seen as desirable and likely (as in expected). At best, the undesirable futures will be mentioned in an accompanying section on risks and their management.

A less myopic approach might be useful, one which would orient the users of the Theory of Change to a more adaptive stance towards the future.

One way forward would be to think of a four-part Theory of Change, each of which has different implications. as follows

The top right cell may already be covered by a Theory of Change. In the desirable but unlikely, and undesirable but likely two cells it would be useful to have ordered lists that describe events, what needs to be done before they happen, and what needs to be done after they happen. In the unlikely and undesirable cell plans for monitoring the status of these events need to be spelled out, and updated on an ongoing basis

We need more doubt and uncertainty!

This week the Swedish Evaluation Society (SVUK)  is holding its annual conference. I took part in a session today on Theories of Change. The first part of my presentation summarised the points I made in a 2018 CEDIL Inception Report titled 'Theories of Change: Technical Challenges with Evaluation Consequences'. Following the presentation I was asked by Gustav Petersson, the discussant, whether we should pay more attention to the process of generating diagrammatic Theories of Change. I could only agree, reflecting that for example it was not uncommon that a representative of a conference working group might summarise a very comprehensive and in-depth discussion in all too brief and succinct terms when reporting back to a plenary. Leaving out, or understating, the uncertainties , ambiguities and disagreements. Similarly the completed version of a diagrammatic Theory of Change is likely to suffer from the same limitations ... being an overly simplified version of a much more complex and nuanced discussions between those involved in its construction that went on beforehand.

Later in the day I was reminded of this section in the Hitchhiker's Guide to the Galaxy where Vroomfondel, representing a group of striking philosophers said '"That's right!" and shouted , "we demand rigidly defined areas of doubt and uncertainty!"

I'm inclined to make a similar kind of request of those developing Theories of Change.  And of those subsequently charged with assessing the evaluability of the associated intervention, including its Theory of Change. What I mean is that the description of the Theory of Change should make it clear which various parts of the theory the owner(s) of that theory are more confident in verses less confident. Along with descriptions of the nature of the doubt or uncertainty and its causes e.g. first-hand experience, or supporting evidence (or lack of) from other sources.

Those undertaking an evaluability assessment could go a step further and convert various specific forms of doubt and uncertainty into evaluation questions that could form an important part of the Terms of Reference for an evaluation.  This might go some way to remedying another problem discussed during the session, which is the all too common (in my experience) phenomena of Terms of Reference only making generic references to an intervention's Theory of Change. For example, by asking in broad terms about "what works and in what circumstances". Rather than the testing of various specific parts of that theory, which would arguably be more useful, and better use of limited time and resources.

The bottom line: The articulation of a Theory of Change should conclude with a list of important evaluation questions. Unless there are good reasons to the contrary, those questions should then appear in the Terms of Reference for a subsequent evaluation

PS: Vroomfondel is a philosopher. He appears in chapter 25 of The Hitchhiker's Guide to the Galaxy, along with his collegue Majikthise, as a representative of the Amalgamated Union of Philosophers, Sages, Luminaries and Other Thinking Persons (AUPSLOTP; the BBC TV version inserts 'Professional' before 'Thinking'). The Union is protesting about Deep Thought, the computer which is being asked to determine the Answer to the Ultimate Question of Life, the Universe and Everything. See https://hitchhikers.fandom.com/wiki/Vroomfondel