Meta-narratives, evaluation and complexity

A meta-narrative is a story about stories. Some evaluations take this form, especially those using participatory approaches to obtain qualitative data from a diversity of sources. Even more conventional expert-led evaluations have an element of storytelling to them as they attempt to weave information obtained from various sources, often opportunistically, into a coherent and plausible overall picture of what happened, and what might happen in future.

Recently I have come across two examples of evaluations that were very much about creating a story about stories. They raised interesting questions about method: how can it be done well? 

Stories about Culture

The first evaluation was of a multiplicity of small arts projects in developing countries, funded by DOEN, a Dutch funding agency. Claudia Fontes used the Most Significant Change technique to elicit and analyse 95 stories from a sample of different kinds of participants in these projects. The aim was to identify what DOEN’s cultural intervention meant to the primary stakeholders. What particularly interested me was one part of the MSC process, which can be a useful step when faced with a large number of stories. This involved the participants categorising the stories into different groupings, according to their commonalities. It was from each of these groupings that the participants then went on to select, through intensive discussion, what they saw as the most significant changes of all. In one country five categories of stories were identified: Personal Development and Growth, Professional Development, Exposure, Change Of Perception And Attitude Towards Art And Artists, and Validation Of Self-Expression. Later on, at the report writing stage, Claudia looked at the contents of these groupings, especially the MSC stories within each, and produced an interpretation of how these groups of stories linked together. In other words, a meta-narrative. 

“For the primary stakeholders in XXXX these categories of change relate to each other in that the personal and professional development of artists and other professionals who support the artists’ work results in a validation of the self-expression of direct (artists) and indirect (public in general) users. This process of affirmation and recovery of ownership of self-expression contributes in turn to a change in society’s perception of art and artists with the potential to make the whole cycle of change sustainable for the sector. Strategies of exposure have a key role in contributing across these changes, and towards the profiling of the sector in general” (italics added) 

In commenting on the report I suggested that in future it might be possible and useful to take a participatory approach to the same task of producing a meta-narrative. Faced with the five groupings (and knowledge of their contents) each participant could be asked to identify expected causal connections between the different groupings, and give some explanation of these views. This can be done through a simple card sorting exercise. The results from multiple participants can then be aggregated, and the result will take the form of a network of relationships between groupings, some being stronger than others (stronger in the sense of more participants’ highlighting a particular causal linkage). This emergent structure can then be visualized using network software. Once visualised in this manner, the structure could be the subject of discussion, and perhaps some revisions.  One important virtue of this kind of process is that it will not necessarily produce a single dominant narrative. Minority and majority views will be discernable. And using network visualization software, the potential complexity would be manageable. Network views can be filtered on multiple variables, such as strength of the causal linkages.

Stories about Conflict

The second evaluation was done by Lundy and McGovern, of Community –based approaches to Post-Conflict “Truth telling” in Northern Ireland” I was sent this and other related papers by Ken Bush, who is exploring methods for evaluating story-telling as a peace building methodology. His draft conceptual framework notes that a “survey of the literature highlighted the lack of an agreed and effective evaluation tool for story-telling in peace-building despite the near universality of the practice and the huge monetary investment by the EU and others in story-telling projects.”

Lundy and McGovern’s paper is a good read, because it explores the many important complications of storytelling in a conflicted society. Not only important issues like appropriate sampling of story tellers, but how the story telling project intentions were framed, and the how the results were presented. The primary product of the project was a publication called “Ardoyne: The Untold Truth”, containing testimonies based on 300 interviews.  The purpose of Lundy and McGovern’s assessment of the project was “to assess the impacts and benefits of community based “truth telling”. This was done by interviewing 50 people from five different stakeholder groups. The results were then written up in their paper.

What we have here is daunting in its complexity: (a) There are the “original” stories, as compiled in the book, (b) then the stories of people’s reactions to these stories and how they were collected and disseminated, (c) and then the authors’ own story about how they collected these stories and  their interpretation of them as a whole. And of course behind all this we have the complex (as colloquially used) context of Northern Ireland!

When reading what might be called Lundy and McGovern’s meta-meta-narrative (i.e. the interpreted results of the interviews) I looked for information on how sources were cited. These are the sorts of phrases I found: “according to respondents”, “many”, ”there was evidence”, “most”, “the vast majority”, “It was felt”, “respondents”, “in the main”, “for many”, “many people”, “there was a very strong opinion”, “it was felt”, “there was a consensus”. “for the majority of participants”, “without exception”, “many interviews”, “overwhelmingly”, “for others”, “some”, “for these respondents”, “one of the most frequently mentioned”, “it was further suggested”, “most respondents”, “the view”, “it was further suggested”, in general respondents were of the view that”, “the experience of those involved…would seem to suggest”, “some respondents”, “the overwhelming majority”, “responses from Union representatives were”, “for some”, “a representative of the community sector”, “that said, others were”, “by another interviewee”. “it was”, and “a significant section of mainly nationalist interviewees”.

I list these here with some hesitation, knowing how often during evaluations I have resorted to using the same vocabulary, when faced with making sense of many different comments by different sources, in a limited period of time. However there are important issues here, made even more important by how often we have to deal with situations like this. How people see things, like their reactions to the Ardoyn stories, matters. How many people see things in a given way matters, who those groups of people are matters, and how the views of different groups overlap also matters. In Lundy and McGovern’s paper we only get glimpses of this kind of underlying social structure. We sometimes get a sense of majority or minority and occasionally which particular group holds a view, and sometimes that a group sharing one view also thinks that…

How could it be done differently? The views of a set of respondents can be summarised in a "two-mode" matrix, with respondents listed in the rows and the descriptions of views listed in columns, and cell values indicating what is known about a person’s views on a listed issue. For example, agreement/disagreement, degree of agreement, or not known. By itself this data is not easy to analyse, other than through frequency counts (e.g. # of people supporting x view, or # views expressed by x person). But it is possible to convert this data into two different kinds of one-mode matrix, showing: (a) how different people are connected to each other (by their shared views) and, (b) how different views are connected to each other (by the same people holding those views). The networks structure of the data in these matrices can be seen and further manipulated using network visualization software

As in many evaluations, Lundy and McGovern were constrained by a confidentiality commitment. Individuals can be anonymised by being categorized into types of people, but this may have its limits if the number of respondents is small and the identity of participants is known to others (if not their specific views). This means the potential to make use of the first kind of network visualization (i.e. a) may be limited, even if the network visualization showed the relationships between types of respondents. However, the second type (i.e. b) should remain an option. To recap, this would show a network of opinions, some strongly linked to many others because they were often shared by the other respondents, others with weaker links to fewer others because they were shared with few, if any, other respondents. The next step would be the development of a narrative, commentary explaining the highlights of the network structure. This would usefully focus on the contents of the different clusters of opinions, and the nature of any bridges between them, especially of the clusters expressed contrasting views.

There are two significant hurdles in front of this approach. Typically not all respondents will express views on all topics, and the number who do will vary across topics. One option would be to filter out the views with the least number of respondents. The other, which I have never tried, would be interesting to explore. That would be to build in a supplementary question in interviews, along the lines of  “…and how many people do you think would feel the same way as you on this issue?”. Their answers would be important in themselves, possibly affecting how the same people might act on their own views. But the same answers could also provide a weighting mechanism for views in an otherwise small sub-sample.

The second hurdle is that the network description of the relationships between the participants views is a snapshot in time. But an evaluation usually requires comparison, with a prior state. This is a problem if the  questions asked by Lundy and McGovern were about current opinions. but if they were about changes in people's views it would not be.

Lets return to the layer below, the stories collected in the original “Ardoyne: The Untold Truth” publications. Stories beget stories. The telling of one can prompt the telling of another. If stories can be seen as linked in this way, then as the number number of stories recounted grows we could end up with a network of stories. Some stories  in that network may be told more often than others, because they are connected to many others, in the minds of the storytellers. These stories might be what complexity science people call "attractors" Although storytellers may start off telling various different stories, their is a likelihood many of them will end up telling this particular story, because of its connectedness, its position in the network.  If these stories are negative, in the sense of provoking antipathy towards others in the same community, then this type of structure may be of concern. Ideally the attractors, the highly connected stories in the network would be positive stories, encouraging peace and cooperation with others. This network structure of stories could be explored by an evaluator asking questions like "What other stories does this story most remind you off? or, "Which of these stories does that story most remind you of?" Or versions thereof. When comparing changes over time the evaluator's focus would then be on the changing contents of the strongly connected versus weakly connected stories.

In this discussion above I have outlined how a network approach could help us construct various types of aggregated (network) views of multiple stories. Because they are built up out of the views of individuals, it would be possible to see where there were varying degrees of agreement within those structures. They would not be biased towards a single (excluding others) narrative, a concern of many people using story-telling approaches including some of the originators of the term meta-narrative

And complex histories

My final comments relate to another form of story-telling, that is grand narrative as done by historians. Yesterday I read with interest Niall Ferguson’s Complexity and Collapse: Empires on the Edge of Chaos(originally in Foreign Affairs). In this article Niall describes the ways some historians have sought to explain the rise and fall in empires, in terms of sequences of events taking place over long periods. In his view they suffer from what Nassim Taleb calls "the narrative fallacy": they construct psychologically satisfying stories on the principle of post hoc, ergo propter hoc ("after this, therefore because of this”). That is, the propensity to over-explain major historical events, to create a long and coherent story where in fact there was none. His alternate view is couched in terms of complexity theory ideas. Given the complexity of modern societies “In reality, the proximate triggers of a crisis are often sufficient to explain the sudden shift from a good equilibrium to a bad mess.” He then qualifies the notion of equilibrium: “a complex economy is characterized by the interaction of dispersed agents, a lack of central control, multiple levels of organization, continual adaptation, incessant creation of new market niches, and the absence of general equilibrium.” Within those systems small changes can have catastrophic (i.e. non-linear) effects, because of the nature of the connectivity involved. Ferguson then goes onto recount examples of the rapidity of decline in some major empires.

One point which he does not make, but which  I think is implicit in his discription of how change can happen in complex systems is that more than one type of small change can trigger the same kind of large scale change. Consider the assissination of Archduke Franz Ferdinand of Austria in Sarajevo in June 1914. Would World War 1 not have happened if that event took place? Not speaking as a historian..my guess is that there are quiet a few other events that could have triggered the start of a war thereafter.

Niall Ferguson complexity based view is in a sense a technocrat’s objection to grand narratives, but perhaps also another kind of grand narrative in its own right. Nevertheless his view does seem to have practical relevance to the writing of evaluation stories: it highlights the need for caution about excessive internal coherence in any story of change and its causes. A network view of causal relationships between types of events, constructed by participants with differing views, might help mitigate against this risk, when it needs to be reduced to a text description.

PS: "In recent years, however, advancements in cognitive neuroscience have suggested that memories unfold across multiple areas of the cortex simultaneously, like a richly interconnected network of stories, rather than an archive of static files." in The Fully Immersive Mind of Oliver Sacks

PS 25 October 2010. Please also see  Networks of self-categorised stories