Reconciling the need for both horizontal and vertical dimensions in a Theory of Change diagram

 

In their usual table form, Logical Frameworks are strong on their horizontal dimension but weak on their vertical dimension. On the horizontal dimension is the explanation of what kind of data will be collected and used to measure the changes that are described. This is good for accountability. On the vertical dimension is the explanation of how events at one level will connect and cause events at another level. This is good for learning.  But unfortunately LogFrames often simply provide lists of events at each level, with relatively little description of which event will connect to which, especially where multiple and mixed sets of connections might be expected between events. On the other hand diagrammatic versions of a Theory of  Change tend to be much better at explicating the various causal pathways at work, but weak on the information they provide on the horizontal dimension - on how various events will be observed and measures. Both of these problems reflect both a lack of space to do both things and different relative priorities pursued within those constraints.

The Donor Committee for Enterprise Development (DCED) has produced a web page based Theory of Change to explain its way of  working, which I think points the way to reconciling these conflicting needs. At first glance here is what you see, when you visit this page of their website.

The different causal pathways are quite visible, more so than within a standard LogFrame table format. But another common weakness of diagrammatic versions of Theories of Change is the lack of explanation of what is going on within each of these pathways. The DCED addressed this problem by allowing visitors to click on a link and be taken to another web page, where visitors get a detailed text description of the available evidence, plus any assumptions, about the causal process(es) that connect the events connected by the arrow.

The one weakness in this DCED ToC diagram is the lack of detail about the horizontal dimension- how the various events described in the diagram will be observed./ measured and by who and when and where. But this is clearly resolvable by using the same approach with the links: enable users to click on any event and be taken to a web page where this information is provided for that specific event. As shown below:

Diversity and complexity? Where should we focus our attention?

 This posting has been promoted by Michael Bamberger's recent two blog postings on "Building complexity into development evaluation" on the 3ie website: Evidence Matters: Towards equitable, inclusive and sustainable development

I started to make some comments underneath each of the two postings but have now decided to try to organise and extend my thoughts here. 

My starting point is an ongoing concern about how unproductive the discussion has been about  complexity (especially in relation to evaluation). Like an elephant giving birth to a mouse, has been my chosen metaphor in the past. There probably is some rhetorical overkill here, but it does express some of my felt frustration with the limited value of the now quite extended discussion.

Michael's blog postings have not allayed these concerns. My concerns start with the idea of measuring complexity, both how you do it and how measuring would in fact be useful. Measuring complexity is Michael's proposed first step in a "practical five-step approach to address complexity-responsive evaluation in a systematic way" A lot of ink has already been spilled on the topic of measurement, which is the first of the five steps. A useful summary can be found in Melanie Mitchel's widely read Complexity: A guided Tour (2009:94-114) and Loyd, 1998, who counted at least 40 different ways. But I cant see any references back to any of these methods, suggesting that not much is being learned from past efforts, which is a pity.  

Along with the challenge of how to do it is the question of why you would want to do it, ...how might it be useful? The second blog posting explains that " In addition to providing stakeholders with an understanding of what complexity means for their program, the checklist also helps decide whether the program is sufficiently complex to merit the additional investment of time and resources required to conduct a complexity-focused evaluation" 

The second of these outcomes might be a more observable consequence, so my first question here is where is the cut-off point in a checklist derived score that would at least inform such a decision, and how is that cut-off point justified. The checklist has 25 attribute questions spread over 4 dimensions. This has not yet been made clear.

My next question is how do the results of this measurement exercise inform the next of the five steps: "Breaking the project into evaluable components and identifying the units of analysis for the component evaluation ". So far, I have not found any answers to this question either. PS 2021 07 29: Michael did reply to a Comment of mine raising the same issue, and suggested that high versus low scores on rated complexity might be one way.

Another concern which I have already written about in my comment on the blog postings is that complexity seems to be very much “in the eye of the beholder”, i.e. depending on who you are and what you are looking for. My partner sees much more complexity in the design and behaviour of moths and butterflies than I do. A friend of mine sees much more complexity in the performance of classical music than I do.  Such observations prompt me to thinking that perhaps we should not put too much effort into trying to objectively measure complexity. Rather, perhaps we should take a more ethnographic perspective on complexity – i.e. we should pay attention to where people are seeing complexity and where they are not, and what are the implications thereof.

If we accept this suggestion it is still the case that the challenge of identifying complexity is still with us, but in a different form. So, I have another suggestion, which is to pay much more attention to diversity, as an important and related concept to complexity. As Scott Page has well described, there is a close and complicated relationship between diversity and complexity. Nevertheless, there are some practically useful points to note about the concept of diversity.  

Firstly the presence of diversity is indicative of the absence of a common constraint, and the presence of many different causal influences. So can be treated as a proxy - indicating the presence of complex processes.  

Secondly, there is extensive and more internally consistent and practically useful set of ways in which diversity can be measured.  These mainly have their origins in the studies of biodiversity but have a much wider applicability.  Diversity can also be measured in other spheres, in human relationships (using social network analysis tools) and how people see the world (using forms of ethnographic enquiry known as pile or card sorting).

Thirdly, diversity has some potentially global relevance as a value and as objective.  Diversity of behaviour can be indicative of choice and empowerment. 

Fourthly, diversity can also be seen as an important independent variable as well, enabling adaptation and creativity. 

All this is not to say that diversity cannot also be problematic.  Some forms of diversity in the present could severely limit the extent of diversity in the future. For example, if within a population there was a wide range of different types of guns held by households and many different views on how and when they could legitimately be used.  At the more mundane level, within organisations different kinds of tasks may benefit from different levels of diversity within the groups addressing those tasks.  So diversity presents a useful and important problematic in a way that the concept of complexity does not. What forms of diversity we want to see, and see sustained over time, and how can they be enabled? Where do we want choice and where should be accept restriction?

Arguing for more attention to diversity, rather than complexity, does not mean there also needs to be a whole new school of evaluation developed around this idea (Get Brand X Evaluation, just hot off the press! Uhh... No). It is consistent with a number of ideas already found useful, including the idea of equifinality (An outcome can arise from multiple different causes) and multifinality ( A cause can have multiple different outcomes), and the idea of multiple conjunctural causation. It is also compatible with a social constructionist and interpretive perspective on social reality.