Friday, October 30, 2009

On the poverty of baselines and targets...

I have been surprised to see how demanding DFID has become on the subject of baseline data. On page 13 of the new DFID Guidance (on using the new formatted Logical Framework) it is stated that ” All projects should have baseline data at all levels before they are approved. In exceptional circumstances, projects may be approved without baseline data at Output level..." Closer to the ground I have witnessed an UK NGO being pressed by DFID-appointed managers of a funding mechanism to deliver the required baseline data. This is despite the fact that the NGO's project will be implemented in a number of countries over a period of years, not all at once.

Meanwhile, in Uganda and Indonesia, I am watching two projects coming to an end. Both had baseline data collected shortly after they started. Neither is showing any signs of intending to do a re-survey at the end of the project period. Is anyone bothered? Not that I can see. Including DFID, who is a donor supporting one of the projects. And in both cases baseline surveys were expensive investments.To make matters worse, in one country the project performance targets were set before the baseline study, and in the other they have never really been agreed on.

I have just completed the final review of one project. We have diligently compared progress made on a set of indicators, against all the original targets. There are of course the usual problems of weak and missing data, and questionable causal links with project interventions. But what bothers me more is how outdated and ill-fitting some of these initial performance measures are. And how little justice this mode of assessment seems to be doing to what the project has been able to do since it started, especially the flexibility of its response in the face of the changing needs of the main partner organisation. Of even greater concern is the fact that this project is being implemented in a large number of districts, in a country that has been going through a significant process of decentralisation. Each district's capacities and needs are different, and not surprisingly the project's activities and results have varied from district to district. There is fact no one single project. Yet our review process, like many others, has in effect treated these district variations as "noise", obscuring what were expected to be region-wide trends over time.

I am now working on some ideas of how to do things differently in my next project review, in the same country. This time the focus will be more on internal comparisons: (a) between locations, (b) between time periods during the project period.

4 comments:

  1. Hello Rick,
    I am surprised to read that this focus on base line data is so much present for output level. So often, measuring outputs is like simple counters that start counting at 0 (so base line data are redundant). At outcome level of course it takes more energy, but maybe that's just a matter of definition. regards, wouter rijneveld

    ReplyDelete
  2. Yes, the DFID guidance does include baseline requirements for Outputs.

    But what Outputs mean is a very open question. In the Indonesian project I am refering to the Outputs were in practice defined as changes taking place within the Govt of Indonesia, not services or goods provided by the assisting international agency to the Govt Of Indonesia. To me it is the latter which should be classed as Outputs, and as you suggest they dont need a baseline, because they by definition start at 0 at the begining of a project

    ReplyDelete
  3. On the other hand, a lot of donors make no effort at all to see that agencies implementing project collect baseline data of any kind. I have seen several projects in the $10 million+ range where considerable original effort was made by the implementing agency and partners to define some intelligent indicators, but no data was ever collected, over a 5-7 year period on the baseline for any of them.

    But the same agencies have also insisted that NGOs with small grants ($25,000) deliver the goods on baseline data.

    I think collecting the data is useful, not just for evaluation purposes at the end of the project, but to test the logic of the project, and the feasibility of data collection. If we have enough intelligent indicators, tested through baseline data collection, then some can be disposed of later, if they don't relate to reality, and new and more relevant indicators added to supplement those that remain.

    Some UN agencies seem to like the idea of "reconstructing baseline data" almost as standard procedure, and while this can be useful where new indicators are developed, I can't see the logic, other than ineptitude, for using it on indicators that were agreed upon - often after considerable discussion - at the beginning of a project.

    So, on balance, if the policy on collecting baseline data is leavened with the possibility of developing new indicators later, I think DFID may be on the right track.

    ReplyDelete
  4. In my experience with projects I have never seen it work either- baselines compared to end of project. But I've seen disappointed farmers- tired of answering the same questions and they don't see much of the actual project.

    I wonder if you can't simply link to national data (already existing) and make it plausible that you made a contribution?

    ReplyDelete