Tuesday, June 05, 2012

Open source evaluation - the way forward?

DFID has set up a special website at projects.dfid.gov.uk where anyone can search for and find details of the development projects it has funded.

As of June this year you can find details of 1512 operational projects, 1767 completed projects and 102 planned projects. The database is updated monthly as a result of an automated trawl through DFID's internal databases. It has been estimated that the database covers 98% of all current projects, with the remaining 2% being omited for security and other reasons.

There are two kinds of search facilities: (a) by key words, (b) by choices from drop down menus [7]. These searches can be combined to narrow a search (in effect, using an AND). But more complex searches using OR or NOT are not yet possible.

The search results are in two forms: (a) A list of projects shown on the webpage, which can also be downloaded as an Excel file. The Excel file has about 30 fields of data, many more than are visible in the webpage listing of the search results; (b) Documents produced by each project, a list of which is viewable after clicking on any project name in a search result. There are 10 different kinds of project documents, ranging from planning documents to progress reports and project completion reports. Evaluation reports are not yet available on this website.

In practice the coverage of project documents is still far from comprehensive. This was my count of what was available in early May, when I was searching for documents relating to DFID's 27 focus countries (out of 103 countries it has worked in up to 2010).

Of all operational projects
Documents available
Up to and including 2010
(27 countries only)
Post-2010 projects
(27 countries only)
Business Case and Intervention Summary
8% (27)
40% (108)
Logical Frameworks

22% (74)
39% (104)
Annual Reviews

17% (55)
9% (24)
Number of  projects
100% (330)
100% (270)

Subject to its continued development, this online database has great potential for enabling what could be called "open source" evaluation, i.e investigation of DFID projects by anyone who can access the website.

With this in mind, I would encourage you to post comments here about:
  •  possible improvements to the database
  •  possible uses of the database.
Re improvements, after having publicised the database on the MandE NEWS email list one respondent made the useful suggestion that the database should include weblinks to existing project websites. Even if the project has closed and the website is no longer maintained access to the website may still be possible through web archive services such as Alexa.com's Wayback machine [PS: It contains copies of www.mande.co.uk dating back to 1998!]

Re uses of the database, earlier today I did a quick analysis of two downloaded  Excel files, one of completed projects and the other of currently operational projects, looking at the proportion of high risk projects in these two groups. The results:

Completed projects
Operational projects

High risk
331 (12%)
677 (19%)

Medium risk
1328 (47%)
1159 (33%)

Low risk
1167 (41%)
1648 (47%)

DFID appears to be funding more high risk projects than in the past. Unfortunately, we dont know what time period the "completed projects" category comes from, or the percentage of projects from that period that are listed in the database. Perhaps this is another possible improvement to the database: Make the source(s) and limitations of the data more visible.

PS 16 June: A salutary blog posting on  "What data can and cannot do" including reminders that,..
  •  Data is not a force unto itself
  •  Data is not a perfect reflection of the world
  •  Data does not speak for itself
  •  Data is not power
  •  Interpreting data is not easy


  1. One comment received by email:
    I hadn't... but response

    Q1: Yes, definitely commend them. I just wish that government would separate out their data from their presentations (not just DFID, it happens all the time). So it's good that it's downloadable, but it'd be better if they had a published data API, and then have their own front-end use that to show the pretty searches, implement the search, draw maps etc. That way, the answer to Q2 is much more likely to be...

    Q2: Yes. I've been a little bit involved with rewired state (rewiredstate.org), doing some hack days where we've been pointed at data (the most recent one I went to was UK Parliament data) and spend a weekend making something fun with it. There's lots of others, I think there are some that are International Development focussed.

    It looks like IATI might be doing this directly, and DFID send them their data.

    Q3. Don't know enough really, although given the work that IATA have done to categorise this stuff it'd be a shame to break that.

  2. Another comment received by email:
    Hi Rick,

    Yes, it is interesting isn't it? I would like to see the DFID site have a click through to the websites of their projects. Certainly all the major ones in Bangladesh have good sites. I've searched my own programme on the DFID site and cannot find a link (admittedly using an iPad in a bumpy car!)