About a month ago I took part in the annual staff conference of a small UK NGO. The focus of the first two days was on identifying the main development issues that NGO should be addressing for the next three years or so. This was part of a wider strategic planning process that was just beginning. During the meeting the CEO made a point of distinguishing the NGO from others by the degree to which its approach was led by the views of its southern partners. If that really was the case, then I think the NGO would have had a justifiable claim to radicalism, something it was well known for in the past. But how could we verify such a claim? As I listened to the ongoing discussion about a range of important global issues, including HIV/AIDS, globalisation, fundamentalism, etc I notice how little, if at all, I could hear of the partner’s views on these issues. That the partners were not physically present in the meeting was not my main worry. I felt the CEO, and other staff, were well aware of the need for appropriate engagement with their partners. What concerned me more was that the discussion about global development issues made no reference to which partner thought what about which issue and why? Assuming the partners did have views, and had been consulted about them in the past, why was there no evidence of that process impacting on how the staff were presenting the development issues in this meeting? I would have thought that citing their partner’s views would have given extra weight to the views being cited. Associated with this concern of mine was a related feeling of far too much ungrounded analysis, that would then be very difficult to convert into a strategy that could be operationalised by the NGO.
The radical alternative would be to focus on the their partner’s views, and to talk explicitly about the areas of agreement and disagreement, both between their partners and with the UK NGO. This is where the NGO has some strategic choices to make (and choices it could fudge). Whose views should it support in future, how and why? And how should it respond to differences between its partners? Where should it seek new partnerships and why? Answers to these questions would help support claims they would like to make about working with, and even being led, by some of their partners. On the other hand, a continuation of talk about global issues without a focus on their partners views on those issues, would suggest they are trying to work through them, simply using them as means to an end.
Monday, April 26, 2004
Friday, April 23, 2004
Monitoring empowerment: A contradiction in terms?
My advice to him was to treat diversity as an indication of empowerment. The rationale for this is spelled out in a conference paper I wrote in 2000, called "Does empowerment start at home? And if so how will we recognise it?". So for any given question about the attitudes or behaviour of the respondents, the survey analyst should examine the range of responses that were given (the SD to be more specific). Not the average response. Here is a quote from that paper:
At the population level, diversity of behaviour can be seen as a gross indicator of agency (of the ability to make choices), relative to homogenous behaviour by the same set of people. Diversity of behaviour suggests there is a range of possibilities that individuals can pursue. At the other extreme is standardisation of behaviour, which we often associate with limited choice. The most notable example being perhaps that of an army. An army is a highly organised structure where individuality is not encouraged, and where standardised and predictable behaviour is very important.
There was an associated footnote, which read:
As noted by some workshop participants, diversity in the behaviour of a set of individuals does not necessarily mean that all have equal choice. Inequalities of power (defined as choice) may still exist. Where we do find diversity in the set as a whole we could then do a more-micro-level analysis and examine the amount of diversity in the behaviour of one individual compared to another.
So, going back to the survey instrument being designed by the multilateral. As well as examining the range of responses to a given question, the researcher should also compare questions in terms of the range of responses to those questions Where is the most and least diversity of responses? Attention might then focus in on the questions with the least range of response. That is where further investigation would be potentially useful, to identify the nature of any common constraints limiting the choices people are making. And if anything can be done to address those common constraints.
What if all the respondents were sending their children to school, does that mean they are not empowered, within this diversity definition of empowerment? That could actually be the case, if there are legal sanctions against not sending children to school. It might also be true that most parents have little real choice about whether to send their children to school. In many developed economies parents are well aware that there are few livelihood options for adults without formal education. This apparently contrary example has some value. Not all forms of lack of empowerment will be of concern to those researching empowerment
Tuesday, April 20, 2004
Why did the chicken cross the road?
On the Euforic website Rob van den Berg considers the challenges facing partners in evaluation....
"Collaborative evaluation is a potential minefield of misunderstandings about definitions, methodologies, concepts, logic and rationalities, reminiscent of the question ‘why did the chicken cross the road?’ The simple answer is that it wanted to get to the other side. Evaluation, however, wants to know whether the chicken took the shortest path across the road; whether it crossed the road in the quickest possible way; whether it did in actual fact reach the other side and whether it expects to remain there; and whether the needs of the chicken have been met by crossing the road..."
I think the case of the chicken who crossed the road has a lot of potential mileage as a metaphor for communicating what people think M&E is all about.
For example, my answer would be:
1. We need to ask the chicken what it had hoped to achieve by crossing the road. Not just pile on the questions regardless of its intentions. What were its objectives or its expectations? Did it in fact have a hypotheses it was going to test? A theory-of-change no less?
2. But we also need to be aware of the possibility that the chicken may have come across some unexpected benefits of crossing the road, after it did so. So just asking about its expectations will not be enough. We also need to ask the chicken about unexpected changes that took place. For example by using the Most Signficant Changes method, we could ask:
"What was the most significant change that took place in your life after you crossed the road?
We need to combine a deductive and theory based approach, with an inductive and experience based approach.
If you think the chicken would disagree with this, or you think there were other stakeholders in the chicken's neghbourhood who would have a different view, let me know, via the MandE NEWS Open Forum
"Collaborative evaluation is a potential minefield of misunderstandings about definitions, methodologies, concepts, logic and rationalities, reminiscent of the question ‘why did the chicken cross the road?’ The simple answer is that it wanted to get to the other side. Evaluation, however, wants to know whether the chicken took the shortest path across the road; whether it crossed the road in the quickest possible way; whether it did in actual fact reach the other side and whether it expects to remain there; and whether the needs of the chicken have been met by crossing the road..."
I think the case of the chicken who crossed the road has a lot of potential mileage as a metaphor for communicating what people think M&E is all about.
For example, my answer would be:
1. We need to ask the chicken what it had hoped to achieve by crossing the road. Not just pile on the questions regardless of its intentions. What were its objectives or its expectations? Did it in fact have a hypotheses it was going to test? A theory-of-change no less?
2. But we also need to be aware of the possibility that the chicken may have come across some unexpected benefits of crossing the road, after it did so. So just asking about its expectations will not be enough. We also need to ask the chicken about unexpected changes that took place. For example by using the Most Signficant Changes method, we could ask:
"What was the most significant change that took place in your life after you crossed the road?
We need to combine a deductive and theory based approach, with an inductive and experience based approach.
If you think the chicken would disagree with this, or you think there were other stakeholders in the chicken's neghbourhood who would have a different view, let me know, via the MandE NEWS Open Forum
Monday, April 12, 2004
Thinking about networks of policies
I have just returned from XXXX in west Africa, where I have been working on PRSP M&E. One of my continuing concerns while there was to get a handle on the complex context in which PRSP M&E activities are taking place. As in most countries the PRSP exists in a complex policy context, it does not stand on its own. It links into, or is expected to link into, a number of other policies and associated implementation processes.
I think the relationship between policies is an area that deserves some serious thinking about, in M&E terms. There are at least two types of relationships that need to be considered:
1. The overlap in objectives of different policy documents.
2. The connection between policies created by information flows between them, once they are implemented and monitored
Policies can overlap in their objectives, this is fairly clear. New policies are often expected to overlap with existing polices. For example, new and specific policies in particular ministries might be expected to help articulate relevant sections of a PRSP. We can measure this overlap in at least two ways: (a) By examining the overlaps in sets of indicators used for M&E of both policies. The PRSP M&E Plan in XXXX has a useful table showing how a number of policies overlap in this respect. (b) By getting owners of two policies to rank the relative importance of their own and the other's policy objectives, and see how their rankings compare. I have done this with a UK NGO, to assess the alignment of a country level strategy with project specific strategies within the same country.
Policies can also be linked by information flows between them, once their implementation begins. In the best case, policy documents are integral parts of high level management cycles. They are plans, which are to be followed by implementation and then hopefully some sort of review processes. And then even more hopefully, some sort of revised policy. In other words, a higher-level version of the project management cycle (i.e. a policy management cycle). Where there are multiple policies, the M&E outputs of one policy management cycle can feed into the planning stages of another policy management cycle. For example, the Annual Progress Report (APR) of the PRSP (Poverty Reduction Strategy Paper) in country XXX is meant to inform the contents of the government's annual budget, as in many other countries. There is now some discussion about when is the best time for the APR to feed into the budget planning discussions, which take place in stages over a period of months.
In some cases there is a relative clear directional nature of these linkages. The APR is expected to influence the budget more than the budget influences the APR. In other cases the net direction of influence is less clear to me, at this stage at least. The World Bank's PRSC (Poverty Reduction Support Credit) includes indicators about the progress made with M&E of the PRSP. The APR should provide evidence of progress made with M&E of the PRSP, and help trigger flow of funds from the WB to the government. But the presence within the PRSC of specific indicators about PRSP M&E capacity may also shape how the PRSP is monitored. Whether it does or not, I have yet to find out.
There are of course many other policies that the PRSP might be expected to influence, and some of those may in turn be expected to influences the PRSP. Somehow these need to be identified and the desired linkages identified. Then we need to know enough about the stages of their respective policy management cycles to identify how and whether the linkages do actually work or not. Without this all the government's efforts put into "communicating the results" of the PRSP begin to look like a shotgun blast into the sky.
Right now I feel we have a very partial and incomplete view of how government and donor policies are and should be interlinking, through exchanges of information between their M&E stages and planning stages. It is the scale that is daunting, including the long cycle times that make them difficult to see as whole processes. The longer the cycle time of any policy management process the less likely that it will function in the same in ways as it was before. Right now we don’t know how the upcoming PRSP review and revision process will look like.
I think the relationship between policies is an area that deserves some serious thinking about, in M&E terms. There are at least two types of relationships that need to be considered:
1. The overlap in objectives of different policy documents.
2. The connection between policies created by information flows between them, once they are implemented and monitored
Policies can overlap in their objectives, this is fairly clear. New policies are often expected to overlap with existing polices. For example, new and specific policies in particular ministries might be expected to help articulate relevant sections of a PRSP. We can measure this overlap in at least two ways: (a) By examining the overlaps in sets of indicators used for M&E of both policies. The PRSP M&E Plan in XXXX has a useful table showing how a number of policies overlap in this respect. (b) By getting owners of two policies to rank the relative importance of their own and the other's policy objectives, and see how their rankings compare. I have done this with a UK NGO, to assess the alignment of a country level strategy with project specific strategies within the same country.
Policies can also be linked by information flows between them, once their implementation begins. In the best case, policy documents are integral parts of high level management cycles. They are plans, which are to be followed by implementation and then hopefully some sort of review processes. And then even more hopefully, some sort of revised policy. In other words, a higher-level version of the project management cycle (i.e. a policy management cycle). Where there are multiple policies, the M&E outputs of one policy management cycle can feed into the planning stages of another policy management cycle. For example, the Annual Progress Report (APR) of the PRSP (Poverty Reduction Strategy Paper) in country XXX is meant to inform the contents of the government's annual budget, as in many other countries. There is now some discussion about when is the best time for the APR to feed into the budget planning discussions, which take place in stages over a period of months.
In some cases there is a relative clear directional nature of these linkages. The APR is expected to influence the budget more than the budget influences the APR. In other cases the net direction of influence is less clear to me, at this stage at least. The World Bank's PRSC (Poverty Reduction Support Credit) includes indicators about the progress made with M&E of the PRSP. The APR should provide evidence of progress made with M&E of the PRSP, and help trigger flow of funds from the WB to the government. But the presence within the PRSC of specific indicators about PRSP M&E capacity may also shape how the PRSP is monitored. Whether it does or not, I have yet to find out.
There are of course many other policies that the PRSP might be expected to influence, and some of those may in turn be expected to influences the PRSP. Somehow these need to be identified and the desired linkages identified. Then we need to know enough about the stages of their respective policy management cycles to identify how and whether the linkages do actually work or not. Without this all the government's efforts put into "communicating the results" of the PRSP begin to look like a shotgun blast into the sky.
Right now I feel we have a very partial and incomplete view of how government and donor policies are and should be interlinking, through exchanges of information between their M&E stages and planning stages. It is the scale that is daunting, including the long cycle times that make them difficult to see as whole processes. The longer the cycle time of any policy management process the less likely that it will function in the same in ways as it was before. Right now we don’t know how the upcoming PRSP review and revision process will look like.
Tuesday, April 06, 2004
Question: How do you assess a country’s ownership of a PRSP?
Answer: Bit by bit. There have been plenty of questions raised about the extent to which PRSP’s are really owned by the government of the country they refer to. (See Google search on ownership of PRSPs). But how do you assess whether a PRSP has country ownership? Well, maybe the way the question is asked could make a difference. One way is to ask who owns what parts of a PRSP. Rather than asking whether the whole document is owned by the whole government. In XXXX there are some PRSP objectives and associated indicators that could easily be adopted and owned by specific sections of government. For example, those relating to education or health, or macro-economic management. Okay, then how would you recognise when sections of government had taken ownership of specific objectives like these? Beyond simply saying so, which may not mean too much, these sections of government might actually collect and make information available about the associated progress indicators. Even stronger ownership might be associated with a detailed analysis of that data, as well as its collection and dissemination. In other words, the section of government would be investing its resources into M&E of their objective, and actually paying a cost in order to enable achievement of that objective. Back in country XXXX, the recently produced Annual Progress Report does not show any signs of any sections of government visibly owning specific sections of the PRSP. Nor is it clear who has been able to provide what information relating to PRSP indicators. In fact there has been an apparent unwillingness to explicitly state what information has not been made available by whom. The scale of lack of ownership has effectively been withheld from view.
Sunday, April 04, 2004
Hypothesis-led Surveys of Influence - on KAP
Today, on my day off, I have read through proposals received from two companies in Bangladesh to do an opinion survey of about 100 people in 20 organisations. The TORs (which I developed) ask a survey company to undertake a two stage survey process:
1. Interview PETRRA (a project) staff about which organisations they think their project has influenced most and least, and in what ways
2. Interview those companies to test out whether PETRRA staff hypotheses about expected influence are supported or not. Using open and closed questions and any other appropriate methods
A hypothesis led impact survey should produced a much more focused bit of evaluation research. The impact of the survey itself should be visible. If it finds the project staff hypotheses not supported by its findings then this should lead to changes, either in how PETRRA understands its influence, or in how the company does such surveys in future.
So far its not looking good. Both proposals have made no mention of the first round of interviews. They are just going to develop some questions of their own and then go out and interview people, and then presumably try to make sense of the results, without the bother of any guiding hypotheses! Oh well... it looks like we will have to provide them with a second round of briefing instructions and hope they get it right this time
Rick
1. Interview PETRRA (a project) staff about which organisations they think their project has influenced most and least, and in what ways
2. Interview those companies to test out whether PETRRA staff hypotheses about expected influence are supported or not. Using open and closed questions and any other appropriate methods
A hypothesis led impact survey should produced a much more focused bit of evaluation research. The impact of the survey itself should be visible. If it finds the project staff hypotheses not supported by its findings then this should lead to changes, either in how PETRRA understands its influence, or in how the company does such surveys in future.
So far its not looking good. Both proposals have made no mention of the first round of interviews. They are just going to develop some questions of their own and then go out and interview people, and then presumably try to make sense of the results, without the bother of any guiding hypotheses! Oh well... it looks like we will have to provide them with a second round of briefing instructions and hope they get it right this time
Rick
Saturday, April 03, 2004
PRSP Monitoring: Target fixation and mission creep
Hi to new and returning visitors to MandE NEWS - from Rick Davies, Editor, MandE NEWS
This is a new step forward by Mande NEWS. I hope by starting up this Blog I might be able to generate some more content for Mande NEWS, on a more continuous basis. This will probably be more ad hoc and more from the hip, so some it will probably end up being deleted, later on in the cold light of day. Anyhow, here goes
Right now I am in XXXX, YYYY, working on monitoring and evaluation of the country's PRS (Poverty Reduction Strategy) Look here for Google findings on PRSPs http://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&q=prsp+poverty+reduction+strategy+paper&btnG=Search
When milestones become millstones: The Annual Progress Report (APR) on the implementation of the PRS is due shortly. The relevant government department is working hard to get it out on time. In the process the end purposes of such an APR are being lost sight of. Getting content is the main concern. Readibility will be a secondary concern, if there is time. Identifying the impact of the APR? Well, there has not yet been time to look at what happened with the last APR yet.
Mission creep at multiple levels in all directions: Donor and other comments are now coming in on the earliest draft of the APR. Could you explain x a bit more...? Why do you have no information on y...?
And this is response to an APR that is already try to track progress relative to indicators not just on the original PRS but at least four other policy documents that have come into the picture since the PRS was written. These include:
- a summary revision created by the government when it came into power
- the Poverty Reduction Support Credit, a WB device
- Multi-Donor Budget Support policy document
- HIPC triggers
- Milleniuum Development Goals (okay, they were there before the PRS)
Needed?: Some continual and public mapping of how the various poverty related (govenment and donor) policies relate to each other (or not), in terms of overlapping indicators and objectives. Both existing and planned policies.
Postcript: 6 hours later, my laptop hard drive leaves this world. The second in 18 months. I will not be buying another HP laptop! Fortunately I have been backing up reasonably often, and I am carrying two memory sticks (much recommended)
This is a new step forward by Mande NEWS. I hope by starting up this Blog I might be able to generate some more content for Mande NEWS, on a more continuous basis. This will probably be more ad hoc and more from the hip, so some it will probably end up being deleted, later on in the cold light of day. Anyhow, here goes
Right now I am in XXXX, YYYY, working on monitoring and evaluation of the country's PRS (Poverty Reduction Strategy) Look here for Google findings on PRSPs http://www.google.com/search?hl=en&lr=&ie=UTF-8&oe=UTF-8&q=prsp+poverty+reduction+strategy+paper&btnG=Search
When milestones become millstones: The Annual Progress Report (APR) on the implementation of the PRS is due shortly. The relevant government department is working hard to get it out on time. In the process the end purposes of such an APR are being lost sight of. Getting content is the main concern. Readibility will be a secondary concern, if there is time. Identifying the impact of the APR? Well, there has not yet been time to look at what happened with the last APR yet.
Mission creep at multiple levels in all directions: Donor and other comments are now coming in on the earliest draft of the APR. Could you explain x a bit more...? Why do you have no information on y...?
And this is response to an APR that is already try to track progress relative to indicators not just on the original PRS but at least four other policy documents that have come into the picture since the PRS was written. These include:
- a summary revision created by the government when it came into power
- the Poverty Reduction Support Credit, a WB device
- Multi-Donor Budget Support policy document
- HIPC triggers
- Milleniuum Development Goals (okay, they were there before the PRS)
Needed?: Some continual and public mapping of how the various poverty related (govenment and donor) policies relate to each other (or not), in terms of overlapping indicators and objectives. Both existing and planned policies.
Postcript: 6 hours later, my laptop hard drive leaves this world. The second in 18 months. I will not be buying another HP laptop! Fortunately I have been backing up reasonably often, and I am carrying two memory sticks (much recommended)
Subscribe to:
Posts (Atom)
