tag:blogger.com,1999:blog-6719829.post1890236059624698450..comments2024-03-27T12:04:05.897+00:00Comments on Rick On the Road: Do we need a Required Level of Failure (RLF)?Rick Davieshttp://www.blogger.com/profile/07028422984421301184noreply@blogger.comBlogger7125tag:blogger.com,1999:blog-6719829.post-5925973160529537382011-08-14T09:12:27.796+00:002011-08-14T09:12:27.796+00:00"If you don't make mistakes, you're n..."If you don't make mistakes, you're not working on hard enough problems. And that's a big mistake." - Frank Wilczek, 2004 Nobel Prize winner in physicsrick davieshttp://www.mande.co.uknoreply@blogger.comtag:blogger.com,1999:blog-6719829.post-1492221792549411832010-10-22T15:17:38.500+00:002010-10-22T15:17:38.500+00:00Thanks Rick and others who have commented so far. ...Thanks Rick and others who have commented so far. As co-author of the project synthesis report Rick mentioned above, and an adviser in DFID's evaluation department, I have a couple of thoughts.<br /><br />First, DFID published a more recent <a href="https://www.dfid.gov.uk/Documents/publications1/evaluation/proj-synth-rpt-ev705.pdf" rel="nofollow">independent review</a> of project completion reports (covering 2005-2008) which may be of interest. This showed the quality of DFID's projects had improved despite taking on riskier projects. It also showed that there was room for improvement (20% of scores were poorly justified), but that there was no evidence of systematic 'marking up' of projects. This review did not focus on looking at the lessons of projects that were found to have been unsuccessful, but I think it would of course make sense for DFID to do this in future studies.<br /><br />Second, I personally think the ideas of a) a minimal level of failure and b) revisiting what counts as project failure are useful ones. We're currently doing some work on improving the way we score projects - and will be able to look into these issues as part of that. <br /><br />On a related note, people may be interested in the growing amount of project level information available on DFID's website through the <a href="http://www.dfid.gov.uk/About-DFID/Finance-and-performance/Project-information/" rel="nofollow">project information database</a> and other <a href="http://www.dfid.gov.uk/ukaid-guarantee" rel="nofollow">commitments to improve aid transparency</a> which will (I believe within the next 12 months) permit others to explore DFID's projects and lessons without having to wait for DFID to undertake such studies. Project reviews are not yet available but will be in 2011.<br /><br />Kerstin Hinds, DFIDKerstin Hindshttp://www.dfid.gov.uknoreply@blogger.comtag:blogger.com,1999:blog-6719829.post-91286344920887666292010-10-12T12:18:48.831+00:002010-10-12T12:18:48.831+00:00I really like this proposal! If we have to have t...I really like this proposal! If we have to have targets let them be for failures (with the caveat that donors tend to construct both failure and success as a moveable policy feast – see Mosse’s Cultivating Development). However, if we understand ‘failure’ as ‘no change’ then the MSC methodology could easily be adapted to determine that.Rosalind Eybennoreply@blogger.comtag:blogger.com,1999:blog-6719829.post-90966134113966315472010-10-11T21:20:55.973+00:002010-10-11T21:20:55.973+00:00Mike Powell writes:
date Mon, Oct 11, 2010 at 6:05...Mike Powell writes:<br />date Mon, Oct 11, 2010 at 6:05 PM<br />subject Re: [MandENEWS] Do we need a Minimum Level of Failure (MLF)?<br /> <br />An interesting blog Rick.<br /><br />It takes a different route to a similar conclusion than Pieter Van Lieshout, lead author of the recent Dutch Research Council report on Dutch development aid. He argues that NGOs should be funded on the basis of a 15% failure rate to counter what he perceives as a lack of innovation in the sector (and also, I think, a lack of critical reflection).<br /><br />For whatever reason some failure may be acceptable, the current refusal to admit or learn from its existence is very damaging. I would argue that RCTs - and other planning, control and evaluative activities which aim to 'prove' success, are even more irrational, desperately seeking to make realities fit bureaucratic and political comnfort zones, rather than accepting their complex nature and developing rational and appropriate, if never complete, methodologies to work with them.<br /><br />Best wishes<br /><br />Mike Powell<br /><br />Director<br />IKM EmergentRick Davieshttps://www.blogger.com/profile/07028422984421301184noreply@blogger.comtag:blogger.com,1999:blog-6719829.post-50273002274458059712010-10-11T17:34:58.272+00:002010-10-11T17:34:58.272+00:00Responding to Pete:
There is some interesting d...Responding to Pete: <br /><br />There is some interesting data on failure within the DFID system, most notably the most recent review of PCR Synthesis Reports in 2005. See “<a href="http://www.dfid.gov.uk/Documents/publications1/evaluation/ev664.pdf" rel="nofollow">An Analysis of Projects and programmes in Prism 2000-2005</a>” by Steve Martin and Kerstin Hinds Corporate Strategy Group, DFID. Some of your skepticism is well founded. It was easy to identify the percentage (68%) of projects “defined as ‘completely’ or ‘largely’ achieving their Goals (Rated 1 or 2)”, but the same degree of specificity about failures was more difficult to come by. Under section “8. Lessons from Project Failures” on page 61 it is stated “There are only 23 projects [out of 453] within the sample that are rated as failing to meet their objectives (i.e. 4 or 5) <em>and which have significant lessons</em>” (italics added). My eyeballing of the bar charts suggests that the percentage with ratings of 4 or 5 was closer to 5%, but perhaps these include those that did not “have significant lessons” <br /><br />More importantly are the 20% or so rated 3 = <em>Likely to be partly achieved</em> (see page 64). It could be argued that those with a rating of 3 should also be included as failures, since their objectives are only likely to be partly achieved, versus largely achieved in the case of rating 2. In other words a successful project should be defined as one likely to achieve more than 50% of its Output and Purpose objectives. Others are failures.<br /><br />Re your concern that “<em>Most projects have elements or components which were successful, and the danger might be that these elements would be lost, sunk with the ship</em>”. This strikes me as a luxury concern. Many of the ostensible lessons from what works in the projects largely achieving their objectives are also being lost sight of. I have recently and direct experience of this.Rick Davieshttps://www.blogger.com/profile/07028422984421301184noreply@blogger.comtag:blogger.com,1999:blog-6719829.post-73841373949307769092010-10-11T16:57:08.298+00:002010-10-11T16:57:08.298+00:00I would wholeheartedly welcome this as a shift in ...I would wholeheartedly welcome this as a shift in the cognitive frameworks with which we talk about international development. The business sector seems to have a healthier relationship with risk in their for-profit endeavours. Yet in the development sector, I've been observing an increasing desperation to “know” what can be inherently beyond logic and induction.<br /><br />The RCT "gold standard" is especially troubling when one is talking about grassroots-up initiatives. Imposing expectations to "try to evaluate every single intervention" on people who are in the process of organizing at the local level is most certainly a drain on their time and scarce resources. And what so many people on the ground have told me again and again is that abstract metrics or research constructs don’t help them understand their relationship to improving the well-being of the people they serve. As members of the community, they read trends through what’s happening on the ground, rather than using any theory. It's time for us to recognize that one can monitor not only through data, but also through dialogue.<br /><br />Let's always consider what is the appropriate cost and complexity needed for evaluation (especially given the size and scope of the program) and aim for proportional expectations so we ensure it remains a tool for learning, not risk-reduction.Jennifer Lentferhttp://www.how-matters.org/noreply@blogger.comtag:blogger.com,1999:blog-6719829.post-49384102988429735142010-10-08T07:41:58.943+00:002010-10-08T07:41:58.943+00:00Interesting. I think one of the problems would ari...Interesting. I think one of the problems would arise in getting any agency to admit to outright project failure (I'm ex-DFID). The ones that work less well always get 'marked up' to avoid loss of face. And where the public purse is involved government agencies are very nervous about PQs and adverse publicity - not least when there is pressure on overall spend, as now. Further to this there are probably very few projects which are a total failuire. Most projects have elements or components which were successful, and the danger might be that these elements would be lost, sunk with the ship.Pete Reidnoreply@blogger.com