Friday, October 30, 2009

On the poverty of baselines and targets...

I have been surprised to see how demanding DFID has become on the subject of baseline data. On page 13 of the new DFID Guidance (on using the new formatted Logical Framework) it is stated that ” All projects should have baseline data at all levels before they are approved. In exceptional circumstances, projects may be approved without baseline data at Output level..." Closer to the ground I have witnessed an UK NGO being pressed by DFID-appointed managers of a funding mechanism to deliver the required baseline data. This is despite the fact that the NGO's project will be implemented in a number of countries over a period of years, not all at once.

Meanwhile, in Uganda and Indonesia, I am watching two projects coming to an end. Both had baseline data collected shortly after they started. Neither is showing any signs of intending to do a re-survey at the end of the project period. Is anyone bothered? Not that I can see. Including DFID, who is a donor supporting one of the projects. And in both cases baseline surveys were expensive investments.To make matters worse, in one country the project performance targets were set before the baseline study, and in the other they have never really been agreed on.

I have just completed the final review of one project. We have diligently compared progress made on a set of indicators, against all the original targets. There are of course the usual problems of weak and missing data, and questionable causal links with project interventions. But what bothers me more is how outdated and ill-fitting some of these initial performance measures are. And how little justice this mode of assessment seems to be doing to what the project has been able to do since it started, especially the flexibility of its response in the face of the changing needs of the main partner organisation. Of even greater concern is the fact that this project is being implemented in a large number of districts, in a country that has been going through a significant process of decentralisation. Each district's capacities and needs are different, and not surprisingly the project's activities and results have varied from district to district. There is fact no one single project. Yet our review process, like many others, has in effect treated these district variations as "noise", obscuring what were expected to be region-wide trends over time.

I am now working on some ideas of how to do things differently in my next project review, in the same country. This time the focus will be more on internal comparisons: (a) between locations, (b) between time periods during the project period.

Tuesday, October 27, 2009

Why we should make economists work harder

"Why we should make life harder for aid agencies" is the title of an article by Tim Harford ("The Undercover Economist") in last weekend's Financial Times magazine section.

I agree with the sentiment, but not with the analysis. I was expecting better, given what I have read of Tim in the past.

Tim's article starts with the problem of how can we, as individual donors, be sure that our aid goes in the right direction and have the expected impact. The next problem, as seen by Tim, is that aid agencies are bureacracies. The solution is competition via a more open market. From within this perspective recent efforts at aid "harmonisation" are viewed by Tim with suspicion, and seen as almost the equivalent to establishing a cartel.

He then asks could agencies be made to compete , not only with each other, but even with private companies, to get funding from donor organisations. And could money (or rather vouchers) be given directly to aid recipients to spend, redeemable for services provided by a range of charities and aid agencies. These ideas he seeas as "radical" and possibly "far fetched" More immediately, he suggests we could "start by asking simple questions about where aid comes from, where it goes, how effective it is and how much is lost to administration – or worse."

I hope Tim will be pleasantly surprised to find his ideas are not seen as radical or far fetched, and in fact have been in play now for quite come time. What Tim really needs to do (apart from more homework before writing articles like this) is to start questioning the assumptions behind his analysis of the nature and benefits of competition amongst aid agencies.

1. In most ordinary markets the purchaser and user are one and the same person. The purchaser and user of aid agency services are different parties, seperated by continents and cultures.

2. In between them is not a single supplier, but a large and complex international aid supply network. See my map of one of the simpler aid supply networks, in a Guardian funded development project in Uganda (map is at the end of the article)

3. The quality of the product/service being provided is much more difficult to assess than that found in many goods and services markets in the UK. Measurement of poverty reduction is a field of its own, improved governance is another order of magnitude more difficult to assess, but nevertheless a common development objective. There are some more measuable oucomes, such as those captured via the Millenium Development Goals e.g. reduced maternal mortality. But these usually require changes in the performance of institutions e.g. national health services. These sorts of change are not simple to measure, let alone achieve. Aid agencies can avoid this challenge by directly supplying health services to poor communities, but they will then fail on another performance metric: sustainability

Tim's idea of vouchers (above) could best be described as quaint. It is now common place for aid agencies in humanitarian emergencies to give cash handouts to families in need, not just vouchers. So they can buy what they need from anyone, not simply "a range of charities and aid agencies" Cash transfers are also being tested for their usefullness in development programmes, where there is no emergency present.

Competition between aid agencies is happening all over the place. DFID has, for years, invited tenders from a wide range of organisations to implement its aid programmes. See their Current Contract Opportunities page But what difference is this making, that is the question. By contracting out work to others DFID moves its own "overhead costs" off its own books, onto others. But the overheads are still there. In fact they are multiplied, because in order to win contracts multiple organsiations invest substantial amounts of time and effort into producing complex documents, but only one wins. Those loosing bids are not products that can easily be sold to other possible buyers, like unused factory stock. Instead the costs of their non-use is figured in to the subsequent bids, including those that win.

So, costs will have gone up, but what about effectiveness? If that has improved, then the increased costs would be justifiable. The problem is, as touched upon above, it is very diffifuclt to measure the effectiveness of many contracted-out projects, because of the scale and complexity of the changes they are trying to achieve.