Two weeks ago I attended a DFID workshop at which Price Waterhouse
Coopers (PwC) consultants presented the results of their work, commissioned by
DFID, on “Monitoring Results from Low Carbon Development”. LCD is one of three
areas of investment by International Climate Fund (ICF). The ICF is “a £2.9bn
financial contribution … provided by the UK Government to support action on
climate change and development. Having started to disperse funds, a
comprehensive results framework is now required to measure the impact of this
investment, to enable learning to inform future programming, and to show value
for money on every pound”
The PwC consultants’ tasks included (a) consultation with
HMG staff on the required functions of the LCD results framework; (b) a detailed
analysis of potentially useful indicators through extensive consultations and research
into the available data; and (c) exploration of opportunities to harmonise
results and/or share methodologies and data collection with others. Their
report documents the large amount of work that has been done, but also acknowledges
that more work is still needed.
Following the workshop I sent in some comments on the PwC
report, some of which I will focus on here because I think they might be of
wider interest. There were three aspects of the PwC proposals that particularly
interested me. One was the fact that they had managed to focus down on 28 indicators,
and were proposing that set be limited even further, down to 20. Secondly, they
had organised the indicators into a LogFrame type structure, but one which is covering two levels
of performance in parallel (within countries and across countries), rather than
in a sequence. Thirdly, they had advocated the use of Multi-Criteria
Analysis (MCA) for the measurement of some of the more complex forms of change
referred to in the Logframe. MCA is similar in structure to the design of weighted checklists, which I have previously discussed here
and elsewhere.
Monitorable versus evaluable frameworks
Monitorable versus evaluable frameworks
As it stands the current LCD LogFrame is a potential means
of monitoring important changes
relating to low carbon development. But it is not yet sufficiently developed to
enable an evaluation of the impact of
efforts aimed at promoting low carbon development. This is because there is not
yet sufficient clarity about the expected causal linkages between the various events
described in the Logframe. It is the case that, as is required by DFID Logframes, weightings have
been given to each of the four Outputs describing their expected impact on
Outcome level changes. But the differences in weightings are modest (+/- 10%) and
each of the Outputs describes a bundle of up to 5 more indicator-specific
changes.
Clarity about the expected causal linkages is an essential “evaluability”
requirement. Impact evaluations in their current form seek to establish not
only what changes occurred, but also their causes. Accounts of causation in
turn need to include not only attribution
(whether A can be said to have caused B) but also explanation (how A caused B). In order for the LCD results
framework to be evaluable, someone needs “connect the dots” in some detail.
That is, identify plausible explanations for how particular indicator-specific
changes are expected influence each other. Once that is done, the LCD program
could be said to have not just a set of indicators of change, but a Theory of Change
about how the changes interact and function as a whole.
Indicator level changes as shared building blocks
Indicator level changes as shared building blocks
There are two subsequent challenges here to developing an
evaluable Theory of Change for LCD. One is the multiplicity of possible causal
linkages. The second is the diversity of perspectives on which of these
possible causal linkages really matter. With 28 different indicator-specific
changes there are, at least hypothetically, many thousands of different possible
combinations that could make up a given Theory of Change (ii, where i
= number of indicator specific changes). But, it can be well argued that “this
is a feature, not a bug”. As the title of this blog suggests, the 28 indicators can be
considered as equivalent to Lego building blocks. The same set (or parts thereof) can be combined
in a multiplicity of ways, to construct very different ToC. The positive
side to this picture is the flexibility and low cost. Different ToC can be constructed
for different countries, but each one does not involve a whole new set of
data collection requirements. In fact it is reasonable to expect that in each
country the causal linkages between different changes may be quite different, because
of the differences in the physical, demographic, cultural and economic context.
Documenting expecting causal linkages (how the blocks are put together)
Documenting expecting causal linkages (how the blocks are put together)
There are other more practical challenges, relating to how
to exploit this flexibility. How do you seek stakeholder views of the expected
causal connections, without getting lost in a sea of possibilities? One
approach I have used in Indonesia and in Vietnam involves the use of simple network
matrices, in workshops involving donor agencies and/or the national partners
associated with a given project. Two examples are shown below. These don’t need
to be understood in detail (one is still in Vietnamese), it is their overall
structure that matters.
A network matrix simply shows the entities that could be
connected in the left column and top row. The convention is that each cell in the matrix provides data
on whether that row entity is connected to that column entity (and it may also
describe the nature of the connection)
The Indonesian example shown below shows expected
relationships between 16 Output indicators (left column) and 11 Purpose level
indicators (top row) in a maternal health project. Workshop participants were
asked to consider one Purpose level indicator at a time, and allocate 100
percentage points across the 16 Output indicators, with more percentage points
= an Output having more expected impact on the Purpose indicator, relative to other Output
indicators. Debate was encouraged between participants as figures were proposed
for each cell down a column. Looking within the matrix we can see that for Purpose
3 it was agreed that Output indicator 1.1 would have the most impact. For some
other Purpose level changes, impact was expected from a wider range of Outputs.
The column on the right side sums up the relative expected impact of each
Output, providing useful guidance on where monitoring attention might be most
usefully focused.
This exercise was completed in little over an hour. The matrix
of results show one set of expected relationships amongst many thousands of other
possible sets that could exist within the same list of indicators. The same
kind of data can be collected on a larger scale via online surveys, where the
options down each column are represented within a single multiple choice
question. Matrices like these, obtained either from different individuals or different
stakeholder groups, can be compared with each other to identify relationships (i.e.
specific cells) where there is the most/least agreement, as well as which relationships
are seen as most important, when all satkeholder views are added up. This information should then inform the
focus of evaluations, allowing scarce attention and resources to be directed to the most critical relationships.
The second example of a network matrix used to explicate a
tacit ToC comes from Vietnam, and is shown below. In this example, a Ministry’s
programmes are shown (unconventionally) across the top row and the country’s 5
year plan objectives are shown down the left column. Cell entries, discussed
and proposed by workshop participants, show the relative expected causal contribution
of each programme to each 5 year plan objective. Summary row on the bottom
shows the aggregate expected contribution of each programme and the summary
column on the right show the aggregate extent to which each 5 year plan objective
was expected to be affected.
Modularity
The modules referred to in the title of this blog can be seen as referring to two types of entities that can be used to construct many different kinds of ToC. One is the indicator-specific changes in the LCD Logframe, for example. By treating them as a standard set available for use by different stakeholders in different settings, we may gain flexibility at a low cost. The other is the grouping of indicator specific changes into categories (e.g. Outputs 1-2-3-4) and larger sets of categories (Outputs, Outcomes, Purpose). The existence of one or more nested types of entities is sometimes described as modularity. In evolutionary theory it has been argued that modularity in design improves evolvablity. This can happen: (a) by allowing specific features to undergo changes without substantially altering the functionality of the entire system, (b) by allowing larger more structural changes to occur by recombining existing functional units.
The modules referred to in the title of this blog can be seen as referring to two types of entities that can be used to construct many different kinds of ToC. One is the indicator-specific changes in the LCD Logframe, for example. By treating them as a standard set available for use by different stakeholders in different settings, we may gain flexibility at a low cost. The other is the grouping of indicator specific changes into categories (e.g. Outputs 1-2-3-4) and larger sets of categories (Outputs, Outcomes, Purpose). The existence of one or more nested types of entities is sometimes described as modularity. In evolutionary theory it has been argued that modularity in design improves evolvablity. This can happen: (a) by allowing specific features to undergo changes without substantially altering the functionality of the entire system, (b) by allowing larger more structural changes to occur by recombining existing functional units.
In the conceptual world of Logframes, and the like, this suggests
that we may need to think of ToC being constructed at multiple levels of detail,
by different sized modules. In the LCD Logframe impact weightings had already been
assigned to each Output, indicating its relative expected contribution to the
Outcomes as a whole. But the flexibility of ToC design at this level was
seriously constrained by the structure of the representational device being
used. In a Logframe Outputs are expected to influence Outcomes, but not the
other way. Nor are they expected to influence each other, contra other more graphic
based logic models. Similarly, both of the above network matrix exercises made
use of existing modules and accepted the kinds of relationship that was expected
between them (Outputs should influence Purpose level changes; Ministry Programmes should influence 5Year Plan objectives achievements).
The value of multiple causal pathways with a ToC
More recently I have seen the ToC for a major area of DFID policy that will be coming under review. This is represented in diagramatic form, showing various kinds of events (including some nested categories of events), and also shows the expected causal relationships between these events. It was quite a complex diagram, perhaps too much so for those who like traffic-light level simplicities. However, what interested me the most is that subsequent versions have been used to show how two specific in-country programs fit within this generic ToC. This has been done by highlighting the particular events that make up one of the number of causal chains that can be found within the generic ToC. In doing so it appears to be successfully addressing a common problem with generic ToC - the inability to reflect the diversity of the programs that make up the policy area described by a generic ToC.
Shared causal pathways justify more evaluation attention
This innovation points to an alternate and additional use of the matrices above. The cell numbers could refer to the numbers of constituent programs in a policy area (and/or which are funded by a single funding mechanism) that involve this particular causal link (i.e. between the row event and the column event). The higher this number, the more important it would be for evaluations to focus on that casual link - because the findings would have relevance across a number of programs in the policy area.
Shared causal pathways justify more evaluation attention
This innovation points to an alternate and additional use of the matrices above. The cell numbers could refer to the numbers of constituent programs in a policy area (and/or which are funded by a single funding mechanism) that involve this particular causal link (i.e. between the row event and the column event). The higher this number, the more important it would be for evaluations to focus on that casual link - because the findings would have relevance across a number of programs in the policy area.