A network approach to the selection of "Most Significant Change" stories

I spent yesterday in a day-long meeting with the staff of an NGO grant-making body, in Ghana. A year ago I had run a two day training workshop for their grantees on the use of the "Most Significant Change" (MSC) method of impact monitoring, a method of monitoring-without-indicators. Since then they had started to collect "Most Significant Change" stories, and they had asked for some feedback on those stories.

In yesterday's meeting, and in my meetings with other organisations in the past, concerns have been expressed about the appropriateness of a hierarchical selection process of MSC stories, when the grantees, and their local partners were all very autonomous organisations, and the last thing the grant making body wanted to do was to create, or reinforce, any view that they were all part of an organisational hierarchy, with the grant making body, and its back donors, at the top.

I explained an alternative way of structuring the selection process, that involved the parallel participation of different stakeholders groups, with a reiterated process of story selection, then feedback to the plenary meeting of all participants. After the meeting yesterday I thought it might be useful to document this alternative, and make it more widely available. So this is what is now available below, in the form of a graphic image of an Excel file. If you click on the image it will be enlarged. Or, click on the link below the image to download the actual Excel file

Your comments and suggestions are invited, please use the Comment facility on this blog.

If you have not heard about MSC before it would be worth looking at the MSC Guide first. Especially section 5 on selection.

Click on the image to make it bigger, or download the Excel file

Postscript: The Washington Post ( 31 Dec 07 online) has an interesting article about how being able to see other people's judgements affects one's own judgements. One of the authors of the study is a well know writer/researcher on networks (Duncan Watts). See also Valdis Krebs' paper "It's the [local] Conversations, Stupid: The link between social interaction and political choice"

Managing expectations about monitoring and evaluation in Katine

Yesterday I went to an event in London, hosted by Barclays, which functioned as the official opening of the Katine project. The Guardian's Katine website went online immediately afterwards, and today's Guardian newspaper features a front page article about the Guardian's involvement in Katine, and a magazine insert giving a detailed description of Katine: the place, the people and the project.

Already some differences in expectations are evident and will need to be managed. Visits to Katine by Guardian and Barclay's staff have clearly had a psychological impact on those staff that visited, and on those they have talked to since. Others are interested to go there as well. But at the same time, AMREF staff have an understandable concern about the manageability of a stream of such visitors. How much of their staff time will be taken up with the planning and hosting of these visits, and what effect will that diversion of resources have on the implementation of the project?

My Terms of Reference (ToRs) already include a responsibility to "Assess whether the Guardian is impacting project delivery or negatively impacting the lives of the community" Already I am thinking that this responsibility needs to be amended to refer to the involvement of the Guardian and Barclays in more general terms, not just media activities.

There are some practical (M&E) steps that could be taken right now. AMREF could start to log the time spent by their staff in planning and hosting each visit by outsiders. On the Guardian and Barclays side, as I suggested to one staff member yesterday, it would be useful if those thinking about a visit could try to be as clear as possible about the objectives of their proposed visit. The nature of what would be a reasonable level of visits is also under negotiation, as part of ongoing contract discussions between AMREF, Barclays and the Guardian.

Another issue that may need to be attended to is the possible impact of the Guardian choosing to focus its media attention on Katine village, which has a population of 1500 people, although AMREF will be working with a much larger group, the 25,000 people living in the wider Katine sub-country (which Katine village is part of). It is possible, though accident and/or intention that a disproportionate amount of project resources may end up being invested in Katine village. For this and other reasons I will need to examine AMREF's plans to see how they intended to address issues of equity: who is being assisted by what project activities, and why so. This leads us into wider issues of what are the most appropriate criteria for assessing AMREF's performance, in addition to equity and effectiveness. This will be the subject of another blog posting, yet to come.

Postscript (31/10/07): I have now set up a Frequently Asked Questions(FAQs) webpage on the topic of Monitoring and Evaluating Success in Katine

Katine: an experiment in more publicly transparent aid processes

Katine is a sub-district of Uganda (map). It is the location of an AMREF development project, funded by the Guardian, and Barclays Bank, starting this year, and scheduled to run for three years. Information about the project will be provided, and regularly updated, on a dedicated Guardian webpage

I will making a number of postings here (on Rick on the Road) and on the Guardian website, about the monitoring and evaluation (M&E) of this project.

At this early stage, there are some identifiable challenges. Some old, some new.
Old ones, which I am already familiar with, will need to be addressed by AMREF in the first instance:

Are the project objectives clear enough to be "evaluable"? Or are they just too fuzzy for anyone to judge? Right now the project staff are engaged in a process of participatory planning with people in Katine. Hopefully this will lead to some more clearly defined objectives, with identifiable and maybe even measurable outcomes, that all agree should be achieved. For example, that 95% of school age girls in the sub-district complete primary school

Amongst the many project activities (relating to education, employment, health and local governance) is there a clear sense of priorities? For example, that improvements in education are most important of all. Without this clarity, it will be hard to weigh up the different achievements and to reach a conclusion about overall success. Ideally the biggest achievements will be in the highest priority areas.

In reality there will be differing views on priorities, and even on the most important expected changes within each area (education, health, etc). Women will probably have different view to men, children will have different views to adults, poorer households will have different views to richer households, etc. Especially within a population of x,000 people. So, the third challenge will be to identify who are the different stakeholders in the project, and how their interests differ. And whose interests should the project prioritise

There are also some new issues, that I will have to address.

AMREF already has staff who are responsible for the monitoring and evaluation of the performance of its projects. But the Guardian and Barclays felt the need for an external M&E person, at least in the earliest stages of this project. The challenge for me is to make my role useful to both parties (AMREF and its two donors) but also to progressively phase out my role , as the Guardian and Barclays gain confidence in AMREF's own capacities to monitor and evaluate its own performance.

Unlike most development aid projects, this project will be in the public eye, via the Guardian, from the beginning. A Ugandan journalist will be based in the community, on a part time basis. The Guardian will be running a blog on the project for three years. There may even be a community run blog, whereby they tell the world, especially the UK, their vew of things. Where possible, project documentation will be made publicly available. All this has risks, as well as great potential for increasing public understanding about how development and aid work (and sometimes doesnt work). The second and much bigger challenge for me here is how to monitor and evaluate the impact of this public exposure

Another challenge, less threatening, will be how to best make use of this major opportunity to communicate with a large number of people. How can we get people to think about development as it happens in real life? Without drowning them in development jargon. And without reinforcing uncritical views about how easy it is to "help people" Perhaps we should start by remembering a quote from Henry Thoreau:

"If I knew for a certainty that a man was coming to my house with the conscious design of doing me good, I should run for my life...for fear that I should get some of his good done to me"

Checklists as mini theories-of-change

During a recent evaluation of a UNICEF assisted health program in Indonesia I was given a copy of a checklist that had been designed for use in assessing the functioning of sub-district health centres in South Sulawesi. You can get a general idea of its structure from the image below. There is a list of attributes of “high performing” health centres down the left, grouped into categories, sub-categories and sub-sub-categories. Down the right side are columns, one for each health centre. Ticks are placed in each row of a column to indicate if the attribute in that row was found in that health centre. I think it is intended that if all the attributes are ticked then the health centre will be deemed to have “graduated” and no longer need to be given development assistance.

While this format has the important virtue of simplicity it does make two assumptions that may be useful to question. It appears that all the listed attributes are essential. This assumes that there is a consensus on what constitutes a “good” health centre. However, in practice, developing that consensus may be an important part of the process of developing a “good” health centre. Not only within the health centre, amongst the staff of the health centre, but also externally, amongst other organisations that the health centre has to work with (e.g. the district hospital, village health posts, and the district health office).

The second assumption is that all the attributes are of equal importance. This seems unlikely. For example, it would be widely agreed that having a supply of oxytocin (attribute 2.4.a) is much more important than “Mother's day celebration implemented each year at sub-district level” (attribute 4.2.b). Attempts to develop the capacity of the health centre will need to be guided by a clear sense of priorities, about what attributes are more important than others. The choice between organising a mothers’ day event and ensuring a supply of oxytocin could be a matter of life or death.

These two “problems” could be seen as opportunities. Attributes on the list could be weighted by asking selected stakeholders to rank the attributes in terms of their relative importance (by allocating points adding to 100 points). If there are a large number of attributes the ranking could start with the major categories, then sub-categories, then attributes within them. Importance could be defined as how much they are likely to continue to improved usage of quality services that will effect people’s health outcomes. The first set of stakeholders could be internal to the health service, and later on external stakeholders could be consulted. Attributes that were given widely different rankings would then be the focus of discussion as to why views varied so much. The assumption here is that this may lead to some convergence of views on priorities. It could also be relevant to staff training agendas. During the evaluation referred to above, we found that comparing different stakeholders ranking of the effectiveness of a number of (other) project activities generated a constructive discussion that increased both stakeholder groups’ understanding of each other, and of the issues involved.

Even when agreement is reached about appropriate weightings a question might be raised about whether this will necessarily lead to expected outcomes. Such as how women are using the health service or their behaviour after visiting the health centre. It would therefore be useful to compare the scores of different health centres and how they related to outcomes observed by those different health centres. How well do these scores predict these outcomes? If they do not, the scores could be re-calculated on the basis of a different set of weightings, to see if emphasising other attributes produced a better fit between health centre scores and observed outcomes of concern. If so, that would suggest the need for a re-orientation of priorities within the health centre. A given set of weightings is in effect the theory-of-change, and the score it generates can be treated as a prediction of an expected outcome. A series of predictions (scores from different health centres) would be needed to see how well the theory fits reality (outcomes observed by those health centres).

Incidentally: A target score on a checklist could be inserted as a single indicator in a Logical Framework, allowing a simple reference to be made to the measurement of a complex outcome. The wider use of checklist scores might help limit the use of overly simplistic indicators of progress, as seen in many Logical Frameworks.

PS: This discussion is not a criticism of the checklist as currently in use. It is an outline of what I think is some of its untapped potential.

Evolving storylines: A participatory design process?

Some years ago...

More than a decade ago, while beginning my PhD, I experimented with the design of a process for evolving stories, through a structured participatory process. The thought was that this could lead to the development of better project designs. A project design should include a theory-of-change, and a theory-of-change when spelled out in detail can be seen as a story. But there could be many different versions of that story, some better than others. If so, then how to discover them?

One possibility was to make use of a Darwinian evolutionary process to search for solutions that have the best fit with their environment. The core of the evolutionary process is the evolutionary algorithm: the re-iteration of processes of variation, selection and retention. The intention was to design a social process that embodied these features. A similar process was later built in as a core feature of the Most Significant Changes (MSC) technique.

I tested the idea out, in a simple and light hearted way, by involving a classroom of secondary students taught by a friend of mine. The environment in which stories would have to develop and survive was that classroom, with its own culture and history. More serious applications could involve the staff of an organisation, and the environment within and around that organisation.

The process:

1. I gave ten of the students some small filing cards, and asked them each to write the beginning of a story on their card, about a student who left school at the end of the year. When completed, these ten cards were then posted, as a column of cards, on the left side of the blackboard, in front of the class. This provided some initial variation
2. I then asked the same students to read all ten cards on the board, and for each of them to identify the story beginning they most liked. This involved selection
3. The students were then asked to each use a second card to write a continuation of the one story beginning they most liked. These story segments were then posted next to the one story beginning they most liked. As a result, some stories beginnings gained multiple new segments, others none. This step involved retention of the selected story beginnings, and introduction of further variation.
4. The students were then asked to look at all the stories again, now they had been extended. I then asked them to write a third generation story segment, which they were to add to the emerging storyline they most liked so far. This process was re-iterated for four generations, until we ran out of class time. A graphic view of the results is shown below (the other being the text of the stories).
   

(left click to magnify image)

Each story segment is represented by one node (circle). Lines connecting the nodes, show which story segment was added to which, forming storylines. In the diagram above the story lines start from the centre and grow outwards. The color of each node represents the identity of the student who wrote that story segment. The size of each node varies according to how many "descendants" it had: how many other story segments were added to it later on. The four concentric circles in the background represent the four generations of the process. PS: Each story segment was only one to three sentences long.

The results:

In evolutionary theory success is defined in minimalist terms, as survival and proliferation. In this exercise three of the initial stories did not survive beyond the first generation (1.7, 1.8, and 1.9). Five others did survive until the fourth generation. Of these two were most prolific (1.6, 1.10), each of which had three descendants by the fourth generation.

Amongst the surviving storylines some were more collective constructions than others. Storylines 1.3 to 1.34, 1.10 to 1.39 and 1.10 to 1.38 had four different contributors (the maximum possible), whereas storylines 1.6 to 1.37 and 1.10 to 1.40 only had two.

As well as analysing the success of different storylines, we can also analyse success at the level of individual participants, using the responses of others as a measure. Individuals varied in the extent to which their story segments were selected by others, and continued by them. One participant's story segments had five continuations by others (see pale brown nodes). At the other extreme, none of the story segments of another participant (see dark green node) were continued by others. Before the exercise I had expected students to favor their own storylines. But as can be seen from the colored nodes in the diagram, this did not happen on a large scale. Some favored their own stories, but most changed storylines at one stage or another.

PS: The results of the process are also amenable to social network analysis. Participants can be seen as linked to each other through their choices of whose stories to select and add on to. It may be useful to test whether there are any coalitions at work. Either those expected prior to the exercise, or ones which were unexpected but important to know about. Within the school students exercise a social network analysis highlighted the presence of one clique of three student, where each added to each other's stories. But two of the students in this clique also added to others stories, and others added to theirs. See network diagram here.

Variations on the process

There are a number of ways in which this process could be varied:

• Vary the extent to which the process facilitator tries to influence the process of evolution . The facilitator could ask all participants to start from one common story beginning in the centre. During the process the facilitators could also introduce events that all storylines must make reference to in one way or another. The facilitator could also choose to specify the some desired characteristics of the end of the story. PS: We could see the facilitator as a representative of the wider / external environment.
  
• Run the process for a longer period. If there were ten generations, or more, it might be possible to find storylines that were built by the contributions of all ten participants. In the wider context it might be of value to find stories that have more collective ownership.
• Allow participants to add two new story segments each, rather than only one. This would increase the amount of variation within the process. But it would also make the process more time consuming. It could be a useful temporary measure to create more variation amongst the stories.
• Limit participation in the process to those whose (initial)storylines had survived so far. This would increase the selection pressure within the process. It could bring the process of evolution to an end (i.e one story remaining).
• Magnify parts of the process. Take two consecutive segments in a story, and re-run the process to start from the first segment, with the aim of reaching the other segment by the n’th generation.
• Introduce a final summary process. At the desired end time ask each participant to priority rank all the surviving storylines. These judgments could then be aggregated to provide a final score for each storyline. (Normally evolutionary processes go on and on, with different “species” emerging and dying out along the way).

How could this process be used for project development purposes?

It could be used at different stages of a project, during planning, implementation or evaluation. At the planning stage it would help think through different scenarios that the project might have to deal with. At the evaluation stage it might provide different versions of the project history, for external evaluators to look at. During implementation it could provide a mix of both scenario analysis and interpretation of history.

The mix of stakeholders involved in the process could be varied, in different ways:

• The participants could be relatively homogenous (e.g. all from same organisation) or more heterogeneous (e.g. from a range of organisations), according to the amount of diversity of storylines that was desired.
• The results of the process generated by one set of stakeholders (e.g. an NGO) could be subject to selection by another (e.g. the NGO's stakeholders). Using the example above, the class teacher could have indicated their preferred storyline from amongst the 10 surviving stories generated by his students.
• It would also be possible to have separate roles for different stakeholders: with one group making the retention decisions (which storlines will be continued) and another making variation decisions (what new story segments to be added on to what storylines (already selected for continuation). The former could be a wide group of stakeholders, and the latter a much smaller group of project planners.

Participants could take on different roles. They could act as themselves or as representatives of specific stakeholders. Responding as individuals may allow participants to think in wider terms than when they are representing their specific stakeholder group. Stakeholder groups could participate via representatives, or as teams (each team making one collective choice about what storylines to continue, and how to do so). A team approach might promote more thought about each step in the evolving storyline, and how the stakeholder group's collective longer term interests could be best served.

At the other extreme, participants’ contributions could be anonymous (but labeled with a pseudonym). This would allow more divergent and risky contributions that might not otherwise appear.

How is this different from scenario planning?

(from Wikipedia) "Scenario development is used in policy planning, organisational development and, generally, when organisations wish to test strategies against uncertain future developments."There are many different ways in which scenario planning is done, but it appears that there are two stages, at least: (a) identification of different scenarios that are of possible concern, (b) identification of means of responding to those scenarios.

Evolving storylines is different in that both processes are interwoven and continuous. Each new story segment is a response to the previous segment, and in turn elaborates the existing scenario (story) in a particular way. It is more adaptive.

In scenario analysis it appears that scenarios are different combinations of circumstances, each of which is seen as potentially important. Such as high inflation and high unemployment. These factors are identified first, prioritised, then used to generate varous combinations. Some of these may not be able to occur together, but others that are become the scenarios. With evolving storylines there no limit on the number or kinds of elements that can be introduced into a story, but there are limits on the number of storylines that can survive.

Scenario analysis seems to be limited to a smaller number of possible outcomes than the storyline process. This may be necessary because the response process is separated from the scenario generation process.

There is also a connection to war games, as applied to the development of corporate strategy development (See Economist, May 31st 2007). These involve competing teams and the taking of turns, "allowing competitors not just to draw up their own strategies but to respond to the choices of others". Evolving storylines could take this process a step further, allowing teams to experiment with multiple parallel strategies. Sometimes a portfolio of approaches may be more useful than a single strategy, not only as a way of managing risk, but also as way of matching the diversity of contexts where an organisation is working. This is especially so for organisations working in multiple countries around the world.

Requests:

• If you have any plans for testing out this process please let me know. I would be happy to provide comments and suggestions: before , during or afterwards.
  
• I would like to develop ways of making this process work with large numbers of participants via the internet, rather than only in face to face meetings. Especially using "open source" processes that could be made freely available via Creative Commons or GNU licenses. If you have any ideas and/or capacity to help with these type of developments please left me know.
  

regards, rick