There is a problem that turns up repeatedly in evaluation practice — and in many other fields — that many of us work around rather than solve directly. The problem is this: the people who know most about a programme, a portfolio, or a set of cases often cannot easily say what they know, or why they make the judgements they do.
This is not a failure of intelligence. It is the normal condition of what Michael Polanyi called tacit knowledge — the "we know more than we can tell" problem that is endemic to any field built on experience and judgement. The challenge for evaluation is to find structured ways of drawing it out.
The theoretical anchor: information as difference
The approach behind The Ethnographic Explorer draws on a deceptively simple idea from Gregory Bateson: information is a difference that makes a difference. What we notice, and consider significant, is always defined by contrast — not by properties of things in isolation. A project is "successful" relative to others; a group is "vulnerable" in comparison to other groups in a given context.
This suggests that a structured method for eliciting knowledge should be built around comparison — specifically, around asking people to identify and articulate differences, and then to explain what those differences imply. The Hierarchical Card Sort methodology is one way of doing this and is the basis of The Ethnographic Explorer's process of inquiry.
How the Ethnographic Explorer works
There are three linked stages.
In the Sort stage, a respondent is presented with a set of cases — projects, organisations, events, beneficiaries, or any entities they know well. They are asked to divide all the cases into two groups representing the most significant difference between them, from their point of view. Each group is named, the difference is recorded, and then the respondent is asked: "What difference does this difference make?" — a question that surfaces the consequence or implication of the distinction. The process repeats on each subgroup until every group contains a single case. The result is a binary tree: a structured, hierarchical map of the respondent's view of the case set.
The tree is informative in three ways: it reveals the contents of the distinctions the respondent considers important; it identifies the limits of their knowledge (where further differences cannot be found); and it indicates the direction of their attention (where further distinctions could usefully be explored).
In the Compare stage, the facilitator asks comparison questions at each split in the tree. Questions can be in degree ("Which group of cases is more likely to face sustainability problems?") or in kind ("How do these two groups differ in terms of their relationship with local government?"). Degree questions produce a ranking of all cases; kind questions produce descriptive contrasts. Both types build on the structure already revealed by the sort.
In the Contrast stage, any two degree-based rankings are plotted against each other in a scatter plot. The resulting quadrant analysis shows where the two rankings agree (cases high on both, or low on both) and where they diverge (cases high on one and low on the other). Adjustable cutoff sliders allow the facilitator to explore different thresholds, and a Spearman correlation coefficient summarises the overall relationship. Outlier cases — those that diverge most between the two rankings — are often the most analytically interesting. Strong relationships can be cast as potentially useful IF...THEN rules
The Ethnographic Explorer
With substantial coding help from Claude AI, I have been developing a browser-based implementation of this methodology — The Ethnographic Explorer — as a standalone single-file application. This new version supersedes an earlier WordPress-based tool at ethnographic.mande.co.uk, which required a server to run. The new version requires nothing beyond a web browser.
▶ Try The Ethnographic Explorer [When you get there, click on Introduction for guidance on how to explore the tool's functions]
The tool is designed for use in a shared-screen video call with a single respondent, or screened to multiple participants in a workshop setting. The facilitator drives the interface; the respondent provides the knowledge. A typical exercise with 8–12 cases takes between 45 minutes and two hours, depending on range of comparisons made.
A worked example: 12 Famous Movies
Two ready-made datasets are available alongside the app to allow you to explore all three stages without setting up an exercise from scratch.
The first — 12 Famous Movies — uses a set of well-known films as cases. The domain framing is an AI-generated perspective on their most significant differences. The sort tree organises the films into groups based on contrasts such as "Grounded in human reality" versus "Mythic and genre-constructed worlds", and further subdivisions exploring the emotional and moral logic of each branch. Two comparison differences are included — Attendance and Age (when produced) — allowing you to explore the Contrast stage and see how these two rankings relate across the twelve films. You can add in other comparisons as needed
The second dataset — 12 Largest Cities in the World — uses major global cities as cases, sorted in terms of their appeal to international tourists. It provides a more applied example, closer to the kind of case sets an evaluator might work with.
To load either dataset, click Import JSON in the top-right toolbar and select the file (already downloaded into your computer).
Download 12 Famous Movies dataset Download 12 Largest Cities dataset
What the tool produces
Each exercise generates a structured record of the respondent's knowledge about a set of cases:
- a binary sort tree, with each split labelled by the most significant difference and its consequence
- one or more named rankings, each derived from a sequence of binary degree judgements working from the root set down through the subsets
- one or more descriptive contrasts, capturing how subgroups differ in kind on specified attributes
- a scatter plot for any pair of degree rankings, with quadrant analysis and Spearman correlation
- a qualitative responses panel, collecting the in-kind descriptions at each split
The full exercise exports as a JSON file, preserving the complete tree structure, all difference descriptions, all responses, and the exercise metadata. Files can be imported to resume exactly where you left off, or shared with a colleague for further analysis.
Beyond evaluation
The underlying structure of the method applies wherever you have a set of cases and a respondent who has differentiated knowledge about them. Some other framings that could be explored with the same tool:
- Organisational learning: a team reviews a set of completed projects, identifying the distinctions that, in retrospect, most predicted success or failure
- Capacity assessment: a trainer sorts a cohort of staff by their readiness for different kinds of work, making the basis for those judgements explicit and discussable
- Stakeholder analysis: a key informant sorts a set of stakeholder organisations, revealing the distinctions they consider most consequential for programme implementation
- Policy analysis: a policy analyst sorts a set of interventions by their perceived effectiveness, then compares that ranking against a ranking of their political feasibility
In each case, the sort structure is the same, the comparison logic is identical, and only the cases, the domain framing, and the named differences change.
An invitation to experiment
The tool is best understood by using it. I would encourage anyone engaged in evaluation, learning, or knowledge management work to try loading in a set of cases they know well — even with a rough sort to start — and see what structure emerges. The Compare and Contrast stages are particularly useful for surfacing assumptions that are rarely made explicit in standard reporting.
I am continuing to develop the tool and would welcome feedback on the methodology, the interface, or applications I have not yet considered.
Accessing the code: In addition to trying the app online, you can download a copy of the code and run it independently. Go to the app in yur directory, right-click, select View Page Source, copy the entire code, paste it into a text file, rename the file to end in .html rather than .txt, and open it in a web browser. The tool runs entirely in your browser — no login required, no data sent anywhere.
Further reading
- Polanyi, M. (1966). The Tacit Dimension. Doubleday. — the source of the "we know more than we can tell" framing.
- Bateson, G. (1972). Steps to an Ecology of Mind. Ballantine Books. — the source of the "difference that makes a difference" framing.
- Kelly, G.A. (1955). The Psychology of Personal Constructs. Norton. — the intellectual precursor to card sorting methods.
- The existing WordPress version of the tool, with documentation and worked examples, is at ethnographic.mande.co.uk
- Help pages for the comparison stage, with nine worked examples of question types, are at the Comparisons Help page
No comments:
Post a Comment