An aid bubble? - Interpreting aid trends

(unattributed source)

This graph (from a DFID presentation) shows what many people have already heard, that the volume of aid given by DFID will continue to increase, but that the amount of money being spent on administering that aid will plateau. Does this divergence mean:

• DFID has discovered a new means of effectively giving development aid that requires less and less administrative overhead each year?
• There is a huge amount of slack capacity within DFID that can safely be pared away for years without hindering its effectiveness?
• This graph is prima facie evidence of an impending aid bubble that is highly likely to burst in the next few years, as one or more mal-administered or corrupted aid programs are publicly exposed, to the discredit of both the good and the bad?
• Yes, it does mean there will be more mal-administered and corrupted aid programs in the future, but not many more people will be worried about this than have been in the past?
• This is a good example of where there is a pressing need for an ex-ante impact assessment of a budgeting strategy ( if ever there was one)?
• The category “Admin budget” is meaningless and in fact the real costs of administering aid have not been adequately disaggregated in this graph.
• Or...?
  

You can record your opinion, by posting a Comment below, or registering your vote on this anonymous opinion poll.

My initial preference would be for the fifth option, even though it is probably unlikely that the results would have much impact on decisions that have already been made.

But on reflection option one may not be so impossible as it seems. DFID may well give more and more of its aid through third parties (multilaterals, and international programmes of different kinds). When it does this those organisations' administration costs will not appear on the DFID books as administration costs, but as aid given. And those organisations can in turn use the same device to manage the apparent levels of their own administration costs, by funding other parties, such as national NGOs.

The cumulative outcome of this re-iterated strategy may well be very perverse, adding up to a bigger proportion of aid being spent on administration than would be the case if the orginal donor had been more directly engaged and been willing to show higher admin costs in its own budget. All this is speculative though. What it does suggest however, is the possible relevance of a "whole supply chain" approach to the evaluation of the costs of different forms of aid. Unlike private sector supply chains, the total cost of delivered aid is not evident in what the beneficiary pays for the final product.

Perhaps these issues could be pursued by the new Independent Advisory Committee on Development Impact?

Aid organisations as self-interested businesses?

This posting has been prompted by a letter I received recently. A client I am working with (evaluating their project and that of another donor) wanted me to sign a confidentiality agreement. While it did not seem excessively restrictive, in terms of general intent it was the very opposite of what I have been trying to encourage this and other donors to do with information about their projects. Increasingly over the past few years I have been pushing for more transparency, not less. The rationale being that the whole aid process would benefit by being more accountable to the public at large, not just to donors or the project manager’s immediate partners and intended beneficiaries. Some of my clients have taken this approach seriously and used their websites to make a whole range of project documents publicly available (See G-rap and PETRRA). Others have agreed in principle but seem to have made little progress in practice.

Parallel to this effort I have been trying to persuade donors and project managers that achieving specific development objectives is not enough For example, increased levels of health service usage, or increased farmers’ incomes. It is also essential that knowledge be accumulated, and made available, about how these objectives were achieved, and what factors made the difference between higher and lower levels of achievement. Without this knowledge the existing achievements are less likely to be sustainable, and they are certainly unlikely to be replicable. Given the scale of most development problems, sustainability and replicability of achievements is absolutely essential. Measuring sustainability and replicability is not easy. But identifying the availability of relevant knowledge should be possible.

If information was made publicly available on how specific developments were achieved then a project can be considered to have created a public good, that others can use. The more usable that knowledge is, the more value that public good is. Businesses do not often do this, though putting usable knowledge in the public domain is becoming more common in the world of software and internet services[1]. Businesses usually have a commercial self interest in keeping secret the key parts of their business processes that would enable others to compete with them in providing the same goods or services. The production of public goods could therefore be seen as a way of differentiating the degree to which aid organisations (of varying kinds) are operating as self-interested businesses versus more public interested organisations. Whether they make and distribute a profit could be considered a secondary matter.

If the production of public goods is accepted as an important defining feature of good aid organisations then more attention to the quality of those goods, and how it could be improved, would be justified. Some might argue that a lot of information products produced by aid organisations are often more like advertising and public relations materials, and better described as “vapourware”[2]. One means of improving the quality of potential public goods would be increased transparency. So we can see not only the final information product (e.g. a book, web page, video, etc), but the drafts and the debate that surrounded their development, and the background data. Not simply as a final package, but during the process. The public could then become engaged, though comment and feedback, in the process of producing the public good(s). This type of semi-open production process is increasingly common in some areas of business (see "Democratizing Innovation", 2005 and Wikipedia entry). In aid organisations this approach could be realised in fairly simple forms through the use of websites to host draft documents, and the use of online open forums and email lists to promote awareness and discussion of those documents. This is not rocket science. But nor is it yet common practice on the scale it should be.

In my argument above transparency has two rationales. One is pragmatic, tranparency could help improve the knowledge that is available about how best to have an impact. On the other hand, when visibly put into practice, transparency may also function as an important signal of intentions, helping us differentiate organisations that are more public interested from those that are more self-interested.

---

[1] For example, in the form of open source products, free internet services and services that inter-operable with those provided by others.

[2] Software products that have a name, and promotional materials, but not much in the way of contents that will actually make them work and deliver what they promise

Social Frameworks: An improvement on the Logical Framework?

Over the last week there have been quite a few email exchanges on the MandE NEWS email list about how to distinguish results from outcomes, results from impact, inputs from outputs, outcomes from impacts, etc. These are the various terms used to describe different levels of a Logical Framework description of a development intervention (in some of the variations of the Logical Framework used by some development agencies). This debate is not new; it comes and goes, and has appeared within most development organisations at some stage or another.

There are two causes of this confusion of nomenclature, in my view. One is that the Logical Framework describes a sequence of causally linked events happening over time. Time flows, it has no natural punctuation marks that can be used to distinguish and categorise stages of a process. It is not possible to “carve nature at the joints” when dealing with time. So any introduction of stage categories like inputs, activities, outputs, outcomes, impacts etc., is artificial, and requires a consensus amongst the users of these terms, if they are to be useful. Within organisations that can be achieved, across organisations it is usually much more difficult.

The second cause is a widespread confusion between two types of hierarchy. The Logical Framework is supposed to represent a temporal hierarchy, of events taking place through time. Here A is supposed to lead to B, which is supposed to lead to C, etc. However some organisations mix in a different kind of hierarchy, when they introduce terms like “components”, and “sub-objectives”. This is a hierarchy of inclusion, where A, B, and C are part of X, and X, Y, and Z are in turn part of some larger entity. So the upper levels of this hierarchy are not the outcomes of lower level activities, but simply wider generalisations or descriptions of types of things described at the lower levels. I have seen this sort of terminology in some UNICEF Logical Frameworks in Indonesia, mixed in with Purpose and Goal statements that are part of a temporal hierarchy.

A Social Framework?

I have been experimenting with an alternative, which does not “throw the baby out with the bathwater”. It could be called a Social Framework, rather than a Logical Framework, because it emphasises people and their relationships, rather than more abstract events and processes.

Let us start with the same tabular structure as the Logical Framework, but then introduce some significant changes. Each row of the narrative column (found on the left side of the Logical Framework) can be used to describe different types of actors (usually organisations or groups, rather than individuals). Actors in adjacent rows will be linked to each other by relationships that already exist, or which will be developed. The overall result is that the table describes a pathway of expected influence, from the actor in the bottom row up to the actor in the top row. The causal mechanisms are the relationships that link the actors. However, as in real life, this process of influence is unlikely to be one-directional. Both parties linked by a relationship may affect each other. For example a UK donor NGO may earn lessons from its southern partner, as well as being an important conduit of funding for that southern partner.

In the Katine project in Uganda this pathway consists of UK donors who fund AMREF who help develop the capacity of local organisations, who provide services to local households. You can see this "pathway to the poor" in the table below. However, you will see I have introduced an extra row, so I can differentiate between the internal workings of AMREF Katine as an organisational actor, and AMREF’s relationships with local organisations. As shown in the second table that follows the first, I could do the same with the other actors in the pathway. But one may not always want this degree of comprehensive detail. Nevertheless, note this basic point: actors and their relationships with each other are the basis of the Social Framework.

Simple version of the pathway

Actor	Local households
Actor	Local organisations
Relationship	AMREF’s relationship with local organisations
Actor	AMREF’s internal functioning
Actor	Donors

More detailed version of the pathway

Actor	Local households
Relationship	Local organisations’ relationship with local households
Actor	Local organisations’ internal functioning
Relationship	AMREF’s relationship with local organisations
Actor	AMREF’s internal functioning
Relationship	Donors’ relationships with AMREF
Actor	Donors’ internal functioning

It is not difficult to see some correspondence between these levels (especially in the first table) and the Logical Framework categories of Inputs, Activities, Outputs, Purpose and Goal. But talking in terms of specific categories of actors is much more tangible and communicable, especially across cultures. So, lets say goodbye to inputs, activities, outputs, etc, for the time being.

Moving on to the next column in the traditional Logical Framework, the Objectively Verifiable Indicators (OVIs), there is no reason why they cannot be used in this more Social Framework. Indicators could be identified for expected internal changes in each actor and for expected changes in their relationships with other adjacent actors in the Social Framework.

Moving on to the next column, the Means of Verificationn (MoV), this column function can also be retained, describing where and how information will be available about the expected changes. In addition, I suggest taking a more social view of this question. The MoV could describe who is expected to know about the changes described by the OVIs in the same row, because of their interests or responsibilities in this area. For example, the household row may have an indicator about households increased access to clean drinking water. In the OVI column in the same row, reference could be made to the Village Water Committee as a body who should know about changes of this type. Their knowledge and then their responses have implications for the sustainability of any improvements in water supply. This actor-oriented view implies the need for participatory approach, built around what people can and should be able to do in the way of monitoring. What is not needed is lists of disembodied items of information that might be found in a report or database somewhere.

Moving on to the next column, in the traditional Logical Framework we normally find Assumptions that refer to other factors that can influence the causal connection between events happening in adjacent rows. In the Social Framework I would suggest that this column describe assumptions about other actors, and the kind of influence that they are expected to have on the actor(s) described in the narrative row of this column (and vice versa).

The work of other NGOs in the same location may involve relationships with some of the actors in the pathways described in the Katine Social Framework. For example, the same government body, or the same community group. This could be flagged by a commentary in the Assumptions column This design flexibility contrasts with the rigidity of nested Logical Frameworks, where it is only possible to represent convergence of plans (Leading to pyramid like structures, with lots of things happening at the base, all converging on a few things at the top).

The Social Framework

This table below is a rough draft of what a Social Framework might look like for part of the Katine project in Uganda. This project is described in detail on the Guardian website.

Narrative description - of expected changes in a pathway of influence	Objectively Verifiable Indicators (OVIs), - evidence of expected change	Means of Verification (MoV), - who should know about the OVIs	Assumptions - about these and other actors
Expected changes in local households	e.g. indicators of access to safe drinking water, children's primary school participation, food sufficiency	e.g. village water committee, school management committee, village administration	e.g. the insurgents will not return again, force relocations and destroy property
Expected changes in local organisations’ relationship with local households	e.g. speed of repair of broken standpipes	e.g. village water committee, village administration	e.g. that local organisations will provide services equitably
Expected changes in local organisations’ internal functioning	e.g scores of weighted checklists re health clinic functioning	e.g. Health Unit Management Committee (HUMC)	e.g. that District Health Service will support implementation of HUMC recommendations
Expected changes in AMREF’s relationship with local organisations			e.g. AMREF will identify other NGOs who are also working with local organisations, cordinate plans with them and learn lessons re those groups
Expected changes in AMREF’s internal functioning			e.g. AMREF HQ will devolve right make public statements re the project
Expected changes in donors’ relationships with AMREF			e.g Existing donors will not prevent AMREF from seeking additional funding from other donors
Expected changes in donors’ internal functioning			e.g. Donors will be able to agree on desired outcomes of their relationship with AMREF

In complex development programmes people have tried to develop hierarchically nested Logical Frameworks, to show how different parts of a complex program connect to each other. But examples of these are not easy to find, despite the fact that there are many complex programmes in existence. In my experience, creating nested Logical Frameworks is not easy, and this may be the explanation for their scarcity.

Connecting up Social Frameworks to describe a more complex picture should be easier, because they have a modular structure. Each row describing an actor is in effect like a building block. These building blocks can be combined in different sequences. So, in addition to the pathway in the table above, a parallel Katine project pathway of influence could be Donors <-> AMREF <-> Ministry of Health <-> District Health Services <-> Local organisations <-> Local Households (actors in italics being part of other influence pathways already documented). This pathway could address the need for a parallel process of policy influencing at the national level, based on AMREF’s experience with local organisations in Katine. This pathway branches off then re-converges with the original pathway (See simple diagram version below)

As noted briefly above, each of the relationships connecting the actors in any part of the pathway is likely to involve two way communications and influence, unlike the one way causality in the Logical Framework. So messages can come back from households, via local organisations to AMREF, then go off to the Ministry of Health. Useful indicators in the AMREF row, could therefore include such developments as improved knowledge about the impact of central government policies on Katine households

PS: I have now updated the ideas describe above in a posting on MandE NEWS called The Social Framework as an alternative to the Logical Framework This is where all future developments of the idea can be found. So, please visit.

Assessing achievements in Katine, Uganda

This weekend I will head off to Uganda for two weeks, to meet the AMREF staff working on the Katine project (See the links on the left side of this blog for more info on this project), and to see Katine sub-country itself, the place and the people. This will be the first of a series of twice-yearly visits that I will be making over the next few years. As part of the preparation for this visit on Monday this week I attended a meeting in London, to go over my Terms of Reference (ToRs) for my visit with staff from AMREF and from the Guardian.

One of the things we discussed was my request last year that AMREF develop a disclosure policy, which will spell what sorts of information they will made publicly available, and under what circumstances. Much to my surprise, that policy has already been developed and approved by the Board in November, but nobody had told me, nor had it’s existence been made public via the AMREF website. This does seem to almost defeat the purpose of the policy, which is unfortunate, since the intentions expressed in the policy do seem positive.

PS: Since that discussion a copy has now been made available on the AMREF website. My questions to you, the reader, are: What do you think of it? How could it be improved? For comparison, here is a similar sort of policy developed by ActionAid.

In the same meeting we also discussed my visit schedule in Uganda. My draft ToRs are here. As you can probably see, the list of things to do is quite long, probably too long to complete in this visit. So my first meeting with AMREF in Uganda will have to focus on prioritising these tasks. Top of my own to-do list is to meet all the AMREF staff in Katine, find out about their various roles, and to talk about their expectations about my role as the external evaluator - what they would and would not like to me doing. I will be bring along all the comments made so far by participants in an online survey of people’s views on this subject, which you can find here online. So far this online survey has focused on a limited number of stakeholders: the staff of AMREF, Guardian and Barclays. But I hope to open it up to wider public participation on return from Uganda. Please feel free to add you views on this subject right now, by commenting on this blog below.

As well as the tasks listed in my Terms of Reference there are many other questions I would like to explore during my visit. Most of these have been prompted by my reading of AMREF’s project documents over the last month, and by reading the Guardian Katine blogs. Here are some of them:

People’s participation: What did the community needs assessments find out about the existence of different community views on development needs in Katine? It is highly unlikely that in a population of 25,000 that they all had the same set of priorities. People’s views are likely to vary by gender, age, and location, at least. How have these views affected the project design?

And in AMREF’s Monitoring and Evaluation Plan for the project, what role will community groups have in monitoring and evaluation of the project? How often will their views be sought? How will those views then feed into decision making about how the project develops? [These questions relate to the equity and relevance dimension of my evaluation work]

Project strategy: Will the project be aiming to assist the whole population evenly, or will it be targeting some groups more than others? Do AMREF have enough staff and financial resources to reach the whole population? Will the various developments in water supply, health and livelihoods be focused at different target groups, or it is essential that a given group of people experience the combined impact of all these developments? How much information is available at this stage about the distribution of the population through the sub-county, and various government services? Could a map of these be made available on the Guardian Katine website, which could be continually updated and unfilled with information, as the project progresses?

Project impact: Where will the impact of the project be most visible in three years time? Will it be in changes in school attendance and completion, changes in people’s health, or changes in their livelihoods? Will the proposed baseline survey enable AMREF to track the changes that are taking place, and separate out the effects of AMREFs inputs, from the effects of other changes taking place in the society and economy? What about unexpected changes that may not have been planned for? How will they be given adequate attention? Is the monitoring and evaluation plan realistic? Is it too ambitious in terms of the information that will be collected?

Sustainability: How will the impact of the assistance provided by AMREF be sustained in the future? Will government be better able, or more willing, to take responsibility for delivering good quality health and education services?

Transparency: What mechanisms does AMREF have for transparency at the local (Katine) level, as distinct from via its website and that of the Guardian? Which of the various project documents produced so far has AMREF made publicly available? What else could be made available right now? What problems, if any, are arising because of this transparency?

If there are other issues you think I should be looking at, please add your comments below.

A network approach to the selection of "Most Significant Change" stories

I spent yesterday in a day-long meeting with the staff of an NGO grant-making body, in Ghana. A year ago I had run a two day training workshop for their grantees on the use of the "Most Significant Change" (MSC) method of impact monitoring, a method of monitoring-without-indicators. Since then they had started to collect "Most Significant Change" stories, and they had asked for some feedback on those stories.

In yesterday's meeting, and in my meetings with other organisations in the past, concerns have been expressed about the appropriateness of a hierarchical selection process of MSC stories, when the grantees, and their local partners were all very autonomous organisations, and the last thing the grant making body wanted to do was to create, or reinforce, any view that they were all part of an organisational hierarchy, with the grant making body, and its back donors, at the top.

I explained an alternative way of structuring the selection process, that involved the parallel participation of different stakeholders groups, with a reiterated process of story selection, then feedback to the plenary meeting of all participants. After the meeting yesterday I thought it might be useful to document this alternative, and make it more widely available. So this is what is now available below, in the form of a graphic image of an Excel file. If you click on the image it will be enlarged. Or, click on the link below the image to download the actual Excel file

Your comments and suggestions are invited, please use the Comment facility on this blog.

If you have not heard about MSC before it would be worth looking at the MSC Guide first. Especially section 5 on selection.

Click on the image to make it bigger, or download the Excel file

Postscript: The Washington Post ( 31 Dec 07 online) has an interesting article about how being able to see other people's judgements affects one's own judgements. One of the authors of the study is a well know writer/researcher on networks (Duncan Watts). See also Valdis Krebs' paper "It's the [local] Conversations, Stupid: The link between social interaction and political choice"

Managing expectations about monitoring and evaluation in Katine

Yesterday I went to an event in London, hosted by Barclays, which functioned as the official opening of the Katine project. The Guardian's Katine website went online immediately afterwards, and today's Guardian newspaper features a front page article about the Guardian's involvement in Katine, and a magazine insert giving a detailed description of Katine: the place, the people and the project.

Already some differences in expectations are evident and will need to be managed. Visits to Katine by Guardian and Barclay's staff have clearly had a psychological impact on those staff that visited, and on those they have talked to since. Others are interested to go there as well. But at the same time, AMREF staff have an understandable concern about the manageability of a stream of such visitors. How much of their staff time will be taken up with the planning and hosting of these visits, and what effect will that diversion of resources have on the implementation of the project?

My Terms of Reference (ToRs) already include a responsibility to "Assess whether the Guardian is impacting project delivery or negatively impacting the lives of the community" Already I am thinking that this responsibility needs to be amended to refer to the involvement of the Guardian and Barclays in more general terms, not just media activities.

There are some practical (M&E) steps that could be taken right now. AMREF could start to log the time spent by their staff in planning and hosting each visit by outsiders. On the Guardian and Barclays side, as I suggested to one staff member yesterday, it would be useful if those thinking about a visit could try to be as clear as possible about the objectives of their proposed visit. The nature of what would be a reasonable level of visits is also under negotiation, as part of ongoing contract discussions between AMREF, Barclays and the Guardian.

Another issue that may need to be attended to is the possible impact of the Guardian choosing to focus its media attention on Katine village, which has a population of 1500 people, although AMREF will be working with a much larger group, the 25,000 people living in the wider Katine sub-country (which Katine village is part of). It is possible, though accident and/or intention that a disproportionate amount of project resources may end up being invested in Katine village. For this and other reasons I will need to examine AMREF's plans to see how they intended to address issues of equity: who is being assisted by what project activities, and why so. This leads us into wider issues of what are the most appropriate criteria for assessing AMREF's performance, in addition to equity and effectiveness. This will be the subject of another blog posting, yet to come.

Postscript (31/10/07): I have now set up a Frequently Asked Questions(FAQs) webpage on the topic of Monitoring and Evaluating Success in Katine

Katine: an experiment in more publicly transparent aid processes

Katine is a sub-district of Uganda (map). It is the location of an AMREF development project, funded by the Guardian, and Barclays Bank, starting this year, and scheduled to run for three years. Information about the project will be provided, and regularly updated, on a dedicated Guardian webpage

I will making a number of postings here (on Rick on the Road) and on the Guardian website, about the monitoring and evaluation (M&E) of this project.

At this early stage, there are some identifiable challenges. Some old, some new.
Old ones, which I am already familiar with, will need to be addressed by AMREF in the first instance:

Are the project objectives clear enough to be "evaluable"? Or are they just too fuzzy for anyone to judge? Right now the project staff are engaged in a process of participatory planning with people in Katine. Hopefully this will lead to some more clearly defined objectives, with identifiable and maybe even measurable outcomes, that all agree should be achieved. For example, that 95% of school age girls in the sub-district complete primary school

Amongst the many project activities (relating to education, employment, health and local governance) is there a clear sense of priorities? For example, that improvements in education are most important of all. Without this clarity, it will be hard to weigh up the different achievements and to reach a conclusion about overall success. Ideally the biggest achievements will be in the highest priority areas.

In reality there will be differing views on priorities, and even on the most important expected changes within each area (education, health, etc). Women will probably have different view to men, children will have different views to adults, poorer households will have different views to richer households, etc. Especially within a population of x,000 people. So, the third challenge will be to identify who are the different stakeholders in the project, and how their interests differ. And whose interests should the project prioritise

There are also some new issues, that I will have to address.

AMREF already has staff who are responsible for the monitoring and evaluation of the performance of its projects. But the Guardian and Barclays felt the need for an external M&E person, at least in the earliest stages of this project. The challenge for me is to make my role useful to both parties (AMREF and its two donors) but also to progressively phase out my role , as the Guardian and Barclays gain confidence in AMREF's own capacities to monitor and evaluate its own performance.

Unlike most development aid projects, this project will be in the public eye, via the Guardian, from the beginning. A Ugandan journalist will be based in the community, on a part time basis. The Guardian will be running a blog on the project for three years. There may even be a community run blog, whereby they tell the world, especially the UK, their vew of things. Where possible, project documentation will be made publicly available. All this has risks, as well as great potential for increasing public understanding about how development and aid work (and sometimes doesnt work). The second and much bigger challenge for me here is how to monitor and evaluate the impact of this public exposure

Another challenge, less threatening, will be how to best make use of this major opportunity to communicate with a large number of people. How can we get people to think about development as it happens in real life? Without drowning them in development jargon. And without reinforcing uncritical views about how easy it is to "help people" Perhaps we should start by remembering a quote from Henry Thoreau:

"If I knew for a certainty that a man was coming to my house with the conscious design of doing me good, I should run for my life...for fear that I should get some of his good done to me"