Comments on the draft DFID evaluation policy

DFID and Independent Advisory Committee on Development Impact have sought public comments on two documents: the Draft Evaluation Policy and the Evaluation Topic List. More information on the public consultation process can be found at MandE NEWS

Comments can be emailed to evaluationfeedback@dfid.gov.uk Here below are two sets of comments that I have sent in:

1. The need for a meta-evaluation of the results of the decentralised evaluation policy

In the List of Potential Evaluation Topics, readers are invited to comment on “any topics you consider very important that we have not listed here”.

One gap which I noted was the lack of any reference to meta-evaluation of the many evaluation activities carried out within the country programmes.

However, the draft Evaluation Policy mentioned above makes eleven references to the role of “decentralised evaluation”. DFID’s decentralised evaluations “are those commissioned by our staff responsible for managing DFID’s programmes, policies and partnerships, normally in collaboration with their development partners”

The references to decentralised evaluations covered the following areas:
- increased use of decentralised evaluation as one of the 4 major priorities for developing the evaluation function in DFID. p.11
- sustaining a strong culture of decentralised evaluation across the Department. p.16
- strengthening its advisory and quality support role for decentralised evaluations p.17
- quality assurance of decentralised evaluations. p.4, p.16
- helping to set standards, providing support and advice, and reporting on quality. p4

But there are no references to a systematic or periodic meta-evaluation of decentralised evaluations. This seems like a major omission. Authority for evaluation has been decentralised, and advisory support and guidance will be provided, but there is no evident complementary mechanism for assessing the results.

PS: meta-evaluations are not the same as synthesis studies. A synthesis study looks at the findings across a number of evaluations, a meta-evaluation looks at the evaluation methods used by a number of evaluations. Most organisations, including DFID, already do quite a few sythesis studies.

2. The need for consultation on evaluation criteria, not just what should be evaluated

There needs to be some debate not just about what is to be evaluated, but on what criteria?

So far, during the present consultation, the question of what to evaluate has been subject of a separate DFID paper (the Evaluation Topic List) but the question of what criteria has only warranted a short section in an annex to the draft policy paper. In that annex DFID list “the internationally-agreed evaluation criteria …[that] will be applied to DFID evaluations. They appropriately note that while “It will not be appropriate to investigate every criterion in depth in every evaluation. DFID evaluators will be requested to provide an explanation of the criteria they have chosen (or not) to cover”. The listed criteria are 1. Relevance, 2. Effectiveness, 3. Effeciency, 4. Impact, 5. Sustainability, 6. Coverage, and 7. Coherence.

Elsewhere on this blog I have argued for the inclusion of two additional criteria to the traditional DAC 5 (1-5 above).These are equity and transparency

It could be argued that criteria 6 (coverage) already covers equity. However the choice of words can be important. Coverage is an apparently technical term, but equity is explicitly about a value: fairness, of process and outcome. DFID’s desire to eliminate of poverty is a statement about values. Values should be clearly stated, not hidden or assumed.

Transparency is not covered at all. Yet transparency is basic to the whole process of evaluation, especially when viewed in a wider context. Without access to information the ability of stakeholders in development programmes to evaluate performance on any of these criteria will be extremely limited. The importance of access to information was emphasised by the United Nations General Assembly in its first session in 1946, which states: “Freedom of information is a fundamental human right and … the touchstone of all the freedoms to which the UN is consecrated.” (Resolution 59)

More recently DFID was one of the founding signatories to the International Aid Transparency Initiative, publicised at the August 2008 High Level Forum on Aid Effectiveness in Accra, Ghana.

Given this recent statement of position by DFID transparency should clearly be included as an evaluation criteria on the DFID list. If this proposal raises concerns about the list becoming too lengthy, one could argue that it should certainly have higher priority than the newly proposed criteria 7 (coherence). In fact, perhaps it should be criteria number 1, ahead of relevance and all other criteria.

An aid bubble? - Interpreting aid trends

(unattributed source)

This graph (from a DFID presentation) shows what many people have already heard, that the volume of aid given by DFID will continue to increase, but that the amount of money being spent on administering that aid will plateau. Does this divergence mean:

• DFID has discovered a new means of effectively giving development aid that requires less and less administrative overhead each year?
• There is a huge amount of slack capacity within DFID that can safely be pared away for years without hindering its effectiveness?
• This graph is prima facie evidence of an impending aid bubble that is highly likely to burst in the next few years, as one or more mal-administered or corrupted aid programs are publicly exposed, to the discredit of both the good and the bad?
• Yes, it does mean there will be more mal-administered and corrupted aid programs in the future, but not many more people will be worried about this than have been in the past?
• This is a good example of where there is a pressing need for an ex-ante impact assessment of a budgeting strategy ( if ever there was one)?
• The category “Admin budget” is meaningless and in fact the real costs of administering aid have not been adequately disaggregated in this graph.
• Or...?
  

You can record your opinion, by posting a Comment below, or registering your vote on this anonymous opinion poll.

My initial preference would be for the fifth option, even though it is probably unlikely that the results would have much impact on decisions that have already been made.

But on reflection option one may not be so impossible as it seems. DFID may well give more and more of its aid through third parties (multilaterals, and international programmes of different kinds). When it does this those organisations' administration costs will not appear on the DFID books as administration costs, but as aid given. And those organisations can in turn use the same device to manage the apparent levels of their own administration costs, by funding other parties, such as national NGOs.

The cumulative outcome of this re-iterated strategy may well be very perverse, adding up to a bigger proportion of aid being spent on administration than would be the case if the orginal donor had been more directly engaged and been willing to show higher admin costs in its own budget. All this is speculative though. What it does suggest however, is the possible relevance of a "whole supply chain" approach to the evaluation of the costs of different forms of aid. Unlike private sector supply chains, the total cost of delivered aid is not evident in what the beneficiary pays for the final product.

Perhaps these issues could be pursued by the new Independent Advisory Committee on Development Impact?

Aid organisations as self-interested businesses?

This posting has been prompted by a letter I received recently. A client I am working with (evaluating their project and that of another donor) wanted me to sign a confidentiality agreement. While it did not seem excessively restrictive, in terms of general intent it was the very opposite of what I have been trying to encourage this and other donors to do with information about their projects. Increasingly over the past few years I have been pushing for more transparency, not less. The rationale being that the whole aid process would benefit by being more accountable to the public at large, not just to donors or the project manager’s immediate partners and intended beneficiaries. Some of my clients have taken this approach seriously and used their websites to make a whole range of project documents publicly available (See G-rap and PETRRA). Others have agreed in principle but seem to have made little progress in practice.

Parallel to this effort I have been trying to persuade donors and project managers that achieving specific development objectives is not enough For example, increased levels of health service usage, or increased farmers’ incomes. It is also essential that knowledge be accumulated, and made available, about how these objectives were achieved, and what factors made the difference between higher and lower levels of achievement. Without this knowledge the existing achievements are less likely to be sustainable, and they are certainly unlikely to be replicable. Given the scale of most development problems, sustainability and replicability of achievements is absolutely essential. Measuring sustainability and replicability is not easy. But identifying the availability of relevant knowledge should be possible.

If information was made publicly available on how specific developments were achieved then a project can be considered to have created a public good, that others can use. The more usable that knowledge is, the more value that public good is. Businesses do not often do this, though putting usable knowledge in the public domain is becoming more common in the world of software and internet services[1]. Businesses usually have a commercial self interest in keeping secret the key parts of their business processes that would enable others to compete with them in providing the same goods or services. The production of public goods could therefore be seen as a way of differentiating the degree to which aid organisations (of varying kinds) are operating as self-interested businesses versus more public interested organisations. Whether they make and distribute a profit could be considered a secondary matter.

If the production of public goods is accepted as an important defining feature of good aid organisations then more attention to the quality of those goods, and how it could be improved, would be justified. Some might argue that a lot of information products produced by aid organisations are often more like advertising and public relations materials, and better described as “vapourware”[2]. One means of improving the quality of potential public goods would be increased transparency. So we can see not only the final information product (e.g. a book, web page, video, etc), but the drafts and the debate that surrounded their development, and the background data. Not simply as a final package, but during the process. The public could then become engaged, though comment and feedback, in the process of producing the public good(s). This type of semi-open production process is increasingly common in some areas of business (see "Democratizing Innovation", 2005 and Wikipedia entry). In aid organisations this approach could be realised in fairly simple forms through the use of websites to host draft documents, and the use of online open forums and email lists to promote awareness and discussion of those documents. This is not rocket science. But nor is it yet common practice on the scale it should be.

In my argument above transparency has two rationales. One is pragmatic, tranparency could help improve the knowledge that is available about how best to have an impact. On the other hand, when visibly put into practice, transparency may also function as an important signal of intentions, helping us differentiate organisations that are more public interested from those that are more self-interested.

---

[1] For example, in the form of open source products, free internet services and services that inter-operable with those provided by others.

[2] Software products that have a name, and promotional materials, but not much in the way of contents that will actually make them work and deliver what they promise

Social Frameworks: An improvement on the Logical Framework?

Over the last week there have been quite a few email exchanges on the MandE NEWS email list about how to distinguish results from outcomes, results from impact, inputs from outputs, outcomes from impacts, etc. These are the various terms used to describe different levels of a Logical Framework description of a development intervention (in some of the variations of the Logical Framework used by some development agencies). This debate is not new; it comes and goes, and has appeared within most development organisations at some stage or another.

There are two causes of this confusion of nomenclature, in my view. One is that the Logical Framework describes a sequence of causally linked events happening over time. Time flows, it has no natural punctuation marks that can be used to distinguish and categorise stages of a process. It is not possible to “carve nature at the joints” when dealing with time. So any introduction of stage categories like inputs, activities, outputs, outcomes, impacts etc., is artificial, and requires a consensus amongst the users of these terms, if they are to be useful. Within organisations that can be achieved, across organisations it is usually much more difficult.

The second cause is a widespread confusion between two types of hierarchy. The Logical Framework is supposed to represent a temporal hierarchy, of events taking place through time. Here A is supposed to lead to B, which is supposed to lead to C, etc. However some organisations mix in a different kind of hierarchy, when they introduce terms like “components”, and “sub-objectives”. This is a hierarchy of inclusion, where A, B, and C are part of X, and X, Y, and Z are in turn part of some larger entity. So the upper levels of this hierarchy are not the outcomes of lower level activities, but simply wider generalisations or descriptions of types of things described at the lower levels. I have seen this sort of terminology in some UNICEF Logical Frameworks in Indonesia, mixed in with Purpose and Goal statements that are part of a temporal hierarchy.

A Social Framework?

I have been experimenting with an alternative, which does not “throw the baby out with the bathwater”. It could be called a Social Framework, rather than a Logical Framework, because it emphasises people and their relationships, rather than more abstract events and processes.

Let us start with the same tabular structure as the Logical Framework, but then introduce some significant changes. Each row of the narrative column (found on the left side of the Logical Framework) can be used to describe different types of actors (usually organisations or groups, rather than individuals). Actors in adjacent rows will be linked to each other by relationships that already exist, or which will be developed. The overall result is that the table describes a pathway of expected influence, from the actor in the bottom row up to the actor in the top row. The causal mechanisms are the relationships that link the actors. However, as in real life, this process of influence is unlikely to be one-directional. Both parties linked by a relationship may affect each other. For example a UK donor NGO may earn lessons from its southern partner, as well as being an important conduit of funding for that southern partner.

In the Katine project in Uganda this pathway consists of UK donors who fund AMREF who help develop the capacity of local organisations, who provide services to local households. You can see this "pathway to the poor" in the table below. However, you will see I have introduced an extra row, so I can differentiate between the internal workings of AMREF Katine as an organisational actor, and AMREF’s relationships with local organisations. As shown in the second table that follows the first, I could do the same with the other actors in the pathway. But one may not always want this degree of comprehensive detail. Nevertheless, note this basic point: actors and their relationships with each other are the basis of the Social Framework.

Simple version of the pathway

Actor	Local households
Actor	Local organisations
Relationship	AMREF’s relationship with local organisations
Actor	AMREF’s internal functioning
Actor	Donors

More detailed version of the pathway

Actor	Local households
Relationship	Local organisations’ relationship with local households
Actor	Local organisations’ internal functioning
Relationship	AMREF’s relationship with local organisations
Actor	AMREF’s internal functioning
Relationship	Donors’ relationships with AMREF
Actor	Donors’ internal functioning

It is not difficult to see some correspondence between these levels (especially in the first table) and the Logical Framework categories of Inputs, Activities, Outputs, Purpose and Goal. But talking in terms of specific categories of actors is much more tangible and communicable, especially across cultures. So, lets say goodbye to inputs, activities, outputs, etc, for the time being.

Moving on to the next column in the traditional Logical Framework, the Objectively Verifiable Indicators (OVIs), there is no reason why they cannot be used in this more Social Framework. Indicators could be identified for expected internal changes in each actor and for expected changes in their relationships with other adjacent actors in the Social Framework.

Moving on to the next column, the Means of Verificationn (MoV), this column function can also be retained, describing where and how information will be available about the expected changes. In addition, I suggest taking a more social view of this question. The MoV could describe who is expected to know about the changes described by the OVIs in the same row, because of their interests or responsibilities in this area. For example, the household row may have an indicator about households increased access to clean drinking water. In the OVI column in the same row, reference could be made to the Village Water Committee as a body who should know about changes of this type. Their knowledge and then their responses have implications for the sustainability of any improvements in water supply. This actor-oriented view implies the need for participatory approach, built around what people can and should be able to do in the way of monitoring. What is not needed is lists of disembodied items of information that might be found in a report or database somewhere.

Moving on to the next column, in the traditional Logical Framework we normally find Assumptions that refer to other factors that can influence the causal connection between events happening in adjacent rows. In the Social Framework I would suggest that this column describe assumptions about other actors, and the kind of influence that they are expected to have on the actor(s) described in the narrative row of this column (and vice versa).

The work of other NGOs in the same location may involve relationships with some of the actors in the pathways described in the Katine Social Framework. For example, the same government body, or the same community group. This could be flagged by a commentary in the Assumptions column This design flexibility contrasts with the rigidity of nested Logical Frameworks, where it is only possible to represent convergence of plans (Leading to pyramid like structures, with lots of things happening at the base, all converging on a few things at the top).

The Social Framework

This table below is a rough draft of what a Social Framework might look like for part of the Katine project in Uganda. This project is described in detail on the Guardian website.

Narrative description - of expected changes in a pathway of influence	Objectively Verifiable Indicators (OVIs), - evidence of expected change	Means of Verification (MoV), - who should know about the OVIs	Assumptions - about these and other actors
Expected changes in local households	e.g. indicators of access to safe drinking water, children's primary school participation, food sufficiency	e.g. village water committee, school management committee, village administration	e.g. the insurgents will not return again, force relocations and destroy property
Expected changes in local organisations’ relationship with local households	e.g. speed of repair of broken standpipes	e.g. village water committee, village administration	e.g. that local organisations will provide services equitably
Expected changes in local organisations’ internal functioning	e.g scores of weighted checklists re health clinic functioning	e.g. Health Unit Management Committee (HUMC)	e.g. that District Health Service will support implementation of HUMC recommendations
Expected changes in AMREF’s relationship with local organisations			e.g. AMREF will identify other NGOs who are also working with local organisations, cordinate plans with them and learn lessons re those groups
Expected changes in AMREF’s internal functioning			e.g. AMREF HQ will devolve right make public statements re the project
Expected changes in donors’ relationships with AMREF			e.g Existing donors will not prevent AMREF from seeking additional funding from other donors
Expected changes in donors’ internal functioning			e.g. Donors will be able to agree on desired outcomes of their relationship with AMREF

In complex development programmes people have tried to develop hierarchically nested Logical Frameworks, to show how different parts of a complex program connect to each other. But examples of these are not easy to find, despite the fact that there are many complex programmes in existence. In my experience, creating nested Logical Frameworks is not easy, and this may be the explanation for their scarcity.

Connecting up Social Frameworks to describe a more complex picture should be easier, because they have a modular structure. Each row describing an actor is in effect like a building block. These building blocks can be combined in different sequences. So, in addition to the pathway in the table above, a parallel Katine project pathway of influence could be Donors <-> AMREF <-> Ministry of Health <-> District Health Services <-> Local organisations <-> Local Households (actors in italics being part of other influence pathways already documented). This pathway could address the need for a parallel process of policy influencing at the national level, based on AMREF’s experience with local organisations in Katine. This pathway branches off then re-converges with the original pathway (See simple diagram version below)

As noted briefly above, each of the relationships connecting the actors in any part of the pathway is likely to involve two way communications and influence, unlike the one way causality in the Logical Framework. So messages can come back from households, via local organisations to AMREF, then go off to the Ministry of Health. Useful indicators in the AMREF row, could therefore include such developments as improved knowledge about the impact of central government policies on Katine households

PS: I have now updated the ideas describe above in a posting on MandE NEWS called The Social Framework as an alternative to the Logical Framework This is where all future developments of the idea can be found. So, please visit.

Assessing achievements in Katine, Uganda

This weekend I will head off to Uganda for two weeks, to meet the AMREF staff working on the Katine project (See the links on the left side of this blog for more info on this project), and to see Katine sub-country itself, the place and the people. This will be the first of a series of twice-yearly visits that I will be making over the next few years. As part of the preparation for this visit on Monday this week I attended a meeting in London, to go over my Terms of Reference (ToRs) for my visit with staff from AMREF and from the Guardian.

One of the things we discussed was my request last year that AMREF develop a disclosure policy, which will spell what sorts of information they will made publicly available, and under what circumstances. Much to my surprise, that policy has already been developed and approved by the Board in November, but nobody had told me, nor had it’s existence been made public via the AMREF website. This does seem to almost defeat the purpose of the policy, which is unfortunate, since the intentions expressed in the policy do seem positive.

PS: Since that discussion a copy has now been made available on the AMREF website. My questions to you, the reader, are: What do you think of it? How could it be improved? For comparison, here is a similar sort of policy developed by ActionAid.

In the same meeting we also discussed my visit schedule in Uganda. My draft ToRs are here. As you can probably see, the list of things to do is quite long, probably too long to complete in this visit. So my first meeting with AMREF in Uganda will have to focus on prioritising these tasks. Top of my own to-do list is to meet all the AMREF staff in Katine, find out about their various roles, and to talk about their expectations about my role as the external evaluator - what they would and would not like to me doing. I will be bring along all the comments made so far by participants in an online survey of people’s views on this subject, which you can find here online. So far this online survey has focused on a limited number of stakeholders: the staff of AMREF, Guardian and Barclays. But I hope to open it up to wider public participation on return from Uganda. Please feel free to add you views on this subject right now, by commenting on this blog below.

As well as the tasks listed in my Terms of Reference there are many other questions I would like to explore during my visit. Most of these have been prompted by my reading of AMREF’s project documents over the last month, and by reading the Guardian Katine blogs. Here are some of them:

People’s participation: What did the community needs assessments find out about the existence of different community views on development needs in Katine? It is highly unlikely that in a population of 25,000 that they all had the same set of priorities. People’s views are likely to vary by gender, age, and location, at least. How have these views affected the project design?

And in AMREF’s Monitoring and Evaluation Plan for the project, what role will community groups have in monitoring and evaluation of the project? How often will their views be sought? How will those views then feed into decision making about how the project develops? [These questions relate to the equity and relevance dimension of my evaluation work]

Project strategy: Will the project be aiming to assist the whole population evenly, or will it be targeting some groups more than others? Do AMREF have enough staff and financial resources to reach the whole population? Will the various developments in water supply, health and livelihoods be focused at different target groups, or it is essential that a given group of people experience the combined impact of all these developments? How much information is available at this stage about the distribution of the population through the sub-county, and various government services? Could a map of these be made available on the Guardian Katine website, which could be continually updated and unfilled with information, as the project progresses?

Project impact: Where will the impact of the project be most visible in three years time? Will it be in changes in school attendance and completion, changes in people’s health, or changes in their livelihoods? Will the proposed baseline survey enable AMREF to track the changes that are taking place, and separate out the effects of AMREFs inputs, from the effects of other changes taking place in the society and economy? What about unexpected changes that may not have been planned for? How will they be given adequate attention? Is the monitoring and evaluation plan realistic? Is it too ambitious in terms of the information that will be collected?

Sustainability: How will the impact of the assistance provided by AMREF be sustained in the future? Will government be better able, or more willing, to take responsibility for delivering good quality health and education services?

Transparency: What mechanisms does AMREF have for transparency at the local (Katine) level, as distinct from via its website and that of the Guardian? Which of the various project documents produced so far has AMREF made publicly available? What else could be made available right now? What problems, if any, are arising because of this transparency?

If there are other issues you think I should be looking at, please add your comments below.