Comments on the draft DFID evaluation policy

DFID and Independent Advisory Committee on Development Impact have sought public comments on two documents: the Draft Evaluation Policy and the Evaluation Topic List. More information on the public consultation process can be found at MandE NEWS

Comments can be emailed to evaluationfeedback@dfid.gov.uk Here below are two sets of comments that I have sent in:

1. The need for a meta-evaluation of the results of the decentralised evaluation policy

In the List of Potential Evaluation Topics, readers are invited to comment on “any topics you consider very important that we have not listed here”.

One gap which I noted was the lack of any reference to meta-evaluation of the many evaluation activities carried out within the country programmes.

However, the draft Evaluation Policy mentioned above makes eleven references to the role of “decentralised evaluation”. DFID’s decentralised evaluations “are those commissioned by our staff responsible for managing DFID’s programmes, policies and partnerships, normally in collaboration with their development partners”

The references to decentralised evaluations covered the following areas:
- increased use of decentralised evaluation as one of the 4 major priorities for developing the evaluation function in DFID. p.11
- sustaining a strong culture of decentralised evaluation across the Department. p.16
- strengthening its advisory and quality support role for decentralised evaluations p.17
- quality assurance of decentralised evaluations. p.4, p.16
- helping to set standards, providing support and advice, and reporting on quality. p4

But there are no references to a systematic or periodic meta-evaluation of decentralised evaluations. This seems like a major omission. Authority for evaluation has been decentralised, and advisory support and guidance will be provided, but there is no evident complementary mechanism for assessing the results.

PS: meta-evaluations are not the same as synthesis studies. A synthesis study looks at the findings across a number of evaluations, a meta-evaluation looks at the evaluation methods used by a number of evaluations. Most organisations, including DFID, already do quite a few sythesis studies.

2. The need for consultation on evaluation criteria, not just what should be evaluated

There needs to be some debate not just about what is to be evaluated, but on what criteria?

So far, during the present consultation, the question of what to evaluate has been subject of a separate DFID paper (the Evaluation Topic List) but the question of what criteria has only warranted a short section in an annex to the draft policy paper. In that annex DFID list “the internationally-agreed evaluation criteria …[that] will be applied to DFID evaluations. They appropriately note that while “It will not be appropriate to investigate every criterion in depth in every evaluation. DFID evaluators will be requested to provide an explanation of the criteria they have chosen (or not) to cover”. The listed criteria are 1. Relevance, 2. Effectiveness, 3. Effeciency, 4. Impact, 5. Sustainability, 6. Coverage, and 7. Coherence.

Elsewhere on this blog I have argued for the inclusion of two additional criteria to the traditional DAC 5 (1-5 above).These are equity and transparency

It could be argued that criteria 6 (coverage) already covers equity. However the choice of words can be important. Coverage is an apparently technical term, but equity is explicitly about a value: fairness, of process and outcome. DFID’s desire to eliminate of poverty is a statement about values. Values should be clearly stated, not hidden or assumed.

Transparency is not covered at all. Yet transparency is basic to the whole process of evaluation, especially when viewed in a wider context. Without access to information the ability of stakeholders in development programmes to evaluate performance on any of these criteria will be extremely limited. The importance of access to information was emphasised by the United Nations General Assembly in its first session in 1946, which states: “Freedom of information is a fundamental human right and … the touchstone of all the freedoms to which the UN is consecrated.” (Resolution 59)

More recently DFID was one of the founding signatories to the International Aid Transparency Initiative, publicised at the August 2008 High Level Forum on Aid Effectiveness in Accra, Ghana.

Given this recent statement of position by DFID transparency should clearly be included as an evaluation criteria on the DFID list. If this proposal raises concerns about the list becoming too lengthy, one could argue that it should certainly have higher priority than the newly proposed criteria 7 (coherence). In fact, perhaps it should be criteria number 1, ahead of relevance and all other criteria.