Enrique Mendizabal has suggested I might like
to comment on the M&E aspects of AusAID’s 'Revitalising Indonesia's Knowledge Sector for
Development Policy' program, discussed on the AusAID’s Engage
blog and Enrique’s On
Think Tanks blog
Along with Enrique, I think the Engage blog posting on the new
Indonesian program is a good development. It would be good to see this
happening at the early stages of other AusAID programs. Perhaps it already has.
Enrique notes that “A
weak point and a huge challenge for the programme is its monitoring and
evaluation. I am afraid I cannot offer much advice on this expect that it
should not be too focused on impact while not paying sufficient attention of
the inputs.
I can only agree. It seems ironic that so much attention is
spent these days on assessing impact, while at the same time most LogFrames no
longer seem to bother to detail the activities level. Yet, in practice the
intervening agencies can reasonably be held most responsible for activities and
outputs and least responsible for impact. It seems like the pendulum of
development agency attention has swung too far, from an undue focus on activities
in the (often mythologised) past to an undue emphasis on impact in the present
Enrique suggests that “A
good alternative is to keep the evaluation of the programme independent from
the delivery of the programme and to look for expert impact evaluators based in
universities (and therefore with a focus on the discipline) to explore options
and develop an appropriate approach for the programme. While the contractor may
manage it it should be under a sub-contract that clearly states the
independence of the evaluators. Having one of the partners in the bid in charge
of the evaluation is only likely to create disincentives towards objectivity.”
This is a complex issue and there is unlikely to be a single
simple solution. In a sense, evaluation has to be part of the work of all the
parties involved, but there need to be checks and balances to make sure this is
being done well. Transparency, in the form of public access to plans and progress
reporting is an important part. Evaluability assessments of what is proposed is
another part. Meta-evaluation and syntheses of what has been done is another.
There will be a need for M&E roles within and outside the management
structures. I agree with Guggenheim and Davis’ (AusAID) comment that “the contractor needs to be involved in
setting the program outcomes alongside AusAID and Bappenas because M&E
begins with clarity on what a program’s objectives are”
Looking at the 2 pages on evaluation in the AusAID design
document (see page 48-9) there are some things new and some things old. The
focus of evaluation as hypothesis testing seems new and is something I have
argued for in
the past, in place of numerous and often vague evaluation questions. On the
other hand the counting of products produced and viewed seems stuck in the
past. Necessary but far from sufficient. What is needed is a more macro-perspective
on change, which might be evident in: (a) changes in the structure of
relationships between the institutional actors involved, and (b) in the
content of the individual relationships. Producing and using products is only
one part of those relationships. The categories of “supply”, “demand”, “intermediaries”
and “enabling environment” are a crude first approximation of what is going on
at the more macro level, which hopefully will soon be articulated into more
detail.
The discussion of the Theory of Change in the annex to the
design document is also interesting. On the one hand the authors rightly argue
that this project and setting is complex and “For complex settings and problems, the ‘causal chain’ model often
applied in service delivery programs is too linear and simplistic for
understanding policy influence” On the other hand, some pages later there
is the inevitable and perhaps mandatory, linear Program Logic Model, with nary
a single feedback loop.
One of the purposes of the ToC (presumably including the Program Logic Model) is to “guide the implementation team to develop a robust monitoring and evaluation system” If so, it seems to me that this would be much easier if the events described in the Program Logic Model were being undertaken by identifiable actors (or categories thereof). However, reading the Program Logic Model we see references to only the broadest categories (government agencies, government policy makers, research organisation and networks of civil society) with one exception – Bappenas.
Both these problems of how to describe complex change processes are
by no means unique to AusAID, they are endemic in aid organisations. Yet at the
same time, all the participants in this discussion I am now part of are
enmeshed in an increasing socially and technologically networked world. We are surrounded
by social networks, but yet seemingly incapable of planning in these terms. As
they say “Go figure”
PS: I also meant to say that I strongly support Enrique’s suggestion
that the ToRs for development projects, and the bids received in response to
those ToRs, should be publicly available online, and used as a means of
generating discussion about what should be done. I think that in the case of DFID at least the ToRs
are already available online, to companies registering as interested in bidding
for aid work. However, open debate is not facilitated and is unlikely to happen
if the only parties present are the companies competing with each other for the
work.
