Over the last few years I have had some involvement with the M&E of three project-funding mechanisms. One in the UK, one in Australia and one in south Asia. In all three cases almost all the thinking about the assessment of performance was focused on the analysis of the individual funded projects, along with some syntheses studies that were designed to make some more aggregate assessment of the results of particular categories of projects. The amount of attention given to the assessment of the project funding mechanism varied from a modest amount to none at all. I think this is almost the reverse of what should be the case.
All funding mechanisms that involve calls for proposals and then use a screening process to assess those proposals have in effect a theory of what makes a viable project. In as much as the people reviewing proposals feel they can rate some proposals as better than others then they probably also have a theory of what makes a good project, and a not so good project. These theories will be in the form of a view of what bundle of attributes, discussed during the review process, make the most difference to how successful a project is, in the short and long term. In an ideal world feedback from project-level monitoring and evaluation activities would lead to refinement of these theories about good projects, and this would be evident in changed selection criteria for accepting and funding project proposals. The funding mechanism would get better and better at spotting and funding good projects. In reality I have never seen this sort of feedback link in operation. At least in explicit form.
There are some broad types of theories that would be well worth testing, because they have some identifiable and significant consequences. One is supported by some prior experience. That is, it has been found that it is not the details of the proposed project activities, but the nature of the implementing partner that is what makes a difference between good and bad, or mediocre project outcomes. If this is true it could prompt a substantial re-weighting of emphasis in many project selection procedures, away from a focus on project activities and towards assessment of the project holding organisation. Another possibility is that there is in fact no significant correlation between how well proposals fit selection criteria and their subsequent performance. One possible response to these findings would be to slim down the project selection procedure and to intensify project monitoring and ongoing capacity building of funded projects.
Wednesday, June 30, 2004
Sunday, June 20, 2004
Treating organisations as though they were machines
The following comments are an excerpt from a response I made to the following paper by Alison Scott(DFID)"Assessing and Monitoring Multilateral Effectiveness", available online here
===============
The multilateral organisation as a machine
19. I was disconcerted to read section 9 on the use of multilateral's own assessment of their effectiveness. Not only about the multilaterals' own lack of capacity to assess their effectiveness, but also the conclusion in para 9.3 that these efforts could not be used, and instead DFID would make its own judgements.
20. When we assess the performance of a machine we ask what is it doing and how does that match against what we expect it to be doing. When we assess the performance of an individual or an organisation, we also ask "what does s/he think they are doing" A person is expected to have agency; to be aware of choices and to make responsible choices. It is that awareness and responsibility which is the foundation of legal judgements that can make the difference between a death sentence, imprisonment, or freedom. On a more mundane level, it is an individual's (or organisation's) knowledge about what has happened which makes the difference between whether what has been done can be changed, avoided in future or replicated. The implication for MEFF is that DFID should be assessing the multilateral's knowledge about what it has been doing, and the effects of what it has been doing. That is what matters.
21. Fortunately, most organisations know more than the sum total of what has been captured by their M&E systems. Knowledge is also captured in other documents, produced by other sections of the organisation. But more importantly, it exists, often in tacit and informal forms, in the heads of people who make decisions about where resources should be allocated.
22. If DFID wants to engage with its multilateral partners, then one means of doing so is by trying to explicate their judgements of their performance, both the criteria they are using, the reasons behind those criteria, and the evidence of achievement on those criteria. This can then be complemented by independent verification by DFID, in the areas of performance that are of the greatest concern. A similar approach was taken with the assessment of a SIDA funded poverty alleviation project in Vietnam (See "A Study Of Perceptions And Responses To Poverty Within The Vietnam-Sweden Mountain Rural Development Programme".
==============
For the full text of my comments on the DFID paper go to http://www.mande.co.uk/wp-content/uploads/2004/MEFFcomments.pdf
===============
The multilateral organisation as a machine
19. I was disconcerted to read section 9 on the use of multilateral's own assessment of their effectiveness. Not only about the multilaterals' own lack of capacity to assess their effectiveness, but also the conclusion in para 9.3 that these efforts could not be used, and instead DFID would make its own judgements.
20. When we assess the performance of a machine we ask what is it doing and how does that match against what we expect it to be doing. When we assess the performance of an individual or an organisation, we also ask "what does s/he think they are doing" A person is expected to have agency; to be aware of choices and to make responsible choices. It is that awareness and responsibility which is the foundation of legal judgements that can make the difference between a death sentence, imprisonment, or freedom. On a more mundane level, it is an individual's (or organisation's) knowledge about what has happened which makes the difference between whether what has been done can be changed, avoided in future or replicated. The implication for MEFF is that DFID should be assessing the multilateral's knowledge about what it has been doing, and the effects of what it has been doing. That is what matters.
21. Fortunately, most organisations know more than the sum total of what has been captured by their M&E systems. Knowledge is also captured in other documents, produced by other sections of the organisation. But more importantly, it exists, often in tacit and informal forms, in the heads of people who make decisions about where resources should be allocated.
22. If DFID wants to engage with its multilateral partners, then one means of doing so is by trying to explicate their judgements of their performance, both the criteria they are using, the reasons behind those criteria, and the evidence of achievement on those criteria. This can then be complemented by independent verification by DFID, in the areas of performance that are of the greatest concern. A similar approach was taken with the assessment of a SIDA funded poverty alleviation project in Vietnam (See "A Study Of Perceptions And Responses To Poverty Within The Vietnam-Sweden Mountain Rural Development Programme".
==============
For the full text of my comments on the DFID paper go to http://www.mande.co.uk/wp-content/uploads/2004/MEFFcomments.pdf
Subscribe to:
Posts (Atom)
