Caveat Emptor: I delegated the writing of this blog posting to Claude AI (Sonnet 4.6), based on an extended prior dialogue on the subject below, then a summary prompt of what was wanted. My post-production edits were quite limited.
--o0o--
There is a situation that turns up repeatedly in evaluation and research practice that is easy to overlook precisely because it looks like a data analysis problem rather than a methodological one. The situation is this: you have a set of cases that have been ranked on multiple factors, along with a ranking of their outcomes, and you want to understand what the relationship is between the factors and the outcome.
This sounds straightforward. And to a degree it is — you can correlate each factor against the outcome, identify the strongest relationships, and build a composite ranking that aggregates them. These are useful things to do. But they share a common assumption that is not always warranted: that the factors combine additively, that each one contributes independently to the outcome, and that a case that scores well on several factors will therefore tend to score well overall.
That assumption is often wrong. And when it is wrong, the gap between your composite ranking and the actual outcomes is telling you something important.
The problem with additive aggregation
Consider a concrete example. You have 63 local authority areas. You have ranked them on ten factors thought to be associated with population-level physical activity — access to green space, deprivation levels, sports facility density, and others. You have also ranked them on an outcome measure. You build a composite ranking from the factors, correlate it with the outcome, and find it predicts reasonably well — perhaps a Spearman r of 0.75.
That is a decent result. But it is hiding something. Some areas with strong factor rankings are performing poorly on the outcome; others with weak factor rankings are performing well. If you look closely, there are two or three quite different combinations of factors that each seem sufficient, on their own, to predict a good outcome. These are not variations on the same story — they are distinct causal pathways. However an additive composite averages across them, and in doing so obscures the structure.
This is what researchers in the QCA tradition call equifinality — multiple routes to the same outcome. Additive methods cannot find it. A decision tree can.
What the Rank Explorer does
The Rank Explorer is a browser-based companion tool to The Ethnographic Explorer. It is designed to import the rankings data that TEE generates from its Contrast tab, though it will also accept ranking data from any other source in the same CSV format.
The tool has four analysis tabs. The first three — Individual Factors, Composite Builder, and Scatter Plot — provide standard additive analysis: Spearman correlations for each factor, composite rankings using several aggregation methods (equal-weight, correlation-weighted, stepwise greedy, and exhaustive search), and a scatter plot that visualises how well your composite ranking predicts the outcome, with adjustable classification thresholds.
The fourth tab, Pathway Explorer, is where the configurational logic comes in. It builds an optimal classification tree over your data using exhaustive search: at each node, every available factor and every possible rank cut-off is tested, and the split that best separates high-outcome from low-outcome cases is chosen. The result is a tree that shows which specific combinations of factor ranks distinguish the two groups, displayed in a row-by-level icicle layout that makes the branching structure easy to follow.
When the gap between additive and configurational results is itself a finding
One of the more useful diagnostics the tool enables is comparing the classification accuracy of the best composite ranking against that of the decision tree. If both are similar, the additive story is probably adequate. If the tree substantially outperforms the composite — reaching, say, 90% or 100% accuracy where the composite only reached 75% — that gap is a finding in itself: the causal structure in the data is better described by conjunctions of conditions than by sums of contributions.
This matters for intervention design. If a high-outcome classification requires both good green space access and good active travel infrastructure (rather than either being substitutable for the other), then improving one without the other may produce no discernible effect. Additive analysis will not surface that conclusion; configurational analysis will.
A note on scale, depth, and selective deepening
The tree-building algorithm uses exhaustive search, which is thorough but computationally intensive. With datasets of 60–70 cases and 10 factors, a depth-3 tree typically builds in a few seconds. Depth 4 or beyond is a different matter: computation time increases steeply, and more importantly, deeper trees on small datasets will often find spurious distinctions — patterns that reflect the quirks of the sample rather than anything real.
The Rank Explorer addresses this through a subgroup analysis feature that may be a modest innovation in decision tree practice for small-N datasets. Once a tree has been built, each leaf node displays an Analyse subgroup → button. Clicking it filters the dataset to only the cases in that leaf and opens a fresh analysis session for that group alone — with all four tabs, including the Pathway Explorer, reconfigured for the subgroup. The outcome cut-off resets automatically to the median of the subgroup's outcome ranks, so the high/low distinction remains balanced within the smaller group.
This allows selective deepening of a specific branch without rebuilding the entire tree at greater depth. If one leaf contains 24 cases that the main tree could not separate further, the subgroup analysis asks a different and analytically legitimate question: within this group, what distinguishes the relatively better-performing cases from the worse ones? The answer applies conditionally — only to cases that reached that leaf — but that conditionality is precisely what makes it interpretable. A banner remains visible throughout the subgroup session as a reminder that the high/low labels are relative to the subgroup, not the full dataset.
For larger or more complex datasets, the stepwise greedy method in the Composite Builder tab is a useful preliminary step: it adds factors to the composite one at a time, selecting whichever remaining factor most improves the correlation with the outcome at each step. The resulting path table shows the marginal contribution of each factor, making it straightforward to identify a smaller subset that carries most of the predictive weight — before running the Pathway Explorer on that reduced set.
Beyond TEE data
The tool is designed as a TEE companion but is not restricted to it. Any CSV with a column of case names and a set of ranking columns will load correctly. Evaluation practitioners who have generated case rankings through other means — expert scoring panels, secondary data, peer comparison exercises — can use the same analytical workflow.
Some framings that could map onto the same tool:
- Programme portfolios: rank a set of projects on design-quality dimensions and an outcome measure, then identify which combinations of design features distinguish the most successful from the rest
- Organisational assessments: rank a set of partner organisations on capability dimensions, use the tree to find which combinations are most predictive of delivery performance
- Cross-country comparison: rank a set of countries or regions on contextual factors alongside a development indicator, and look for the configurational patterns that additive index approaches miss
In each case the structure is the same: cases, factor rankings, an outcome ranking, and the question of what the relationship looks like once you stop assuming it is additive.
An invitation to experiment
The tool is best explored with data you already have. If you have ever built a composite index and felt that it was not quite capturing something you could see in the data, or have had the experience of an outlier case that your model consistently misclassifies, the Pathway Explorer is a reasonable next step. Loading your own data, building a tree at depth 2 or 3, and comparing the pathway classification against your composite should take no more than a few minutes.
I am continuing to develop both tools and would welcome feedback on the approach, the interface, or uses I have not considered.
Accessing the code: The Rank Explorer runs entirely in your browser — no login required, no data is transmitted anywhere. To save your own copy of the code, open the tool, right-click, select View Page Source, copy the entire code, paste it into a text file, rename it to end in .html rather than .txt, and open it in any web browser.
Further reading
- Schneider, C. Q., & Wagemann, C. (2012). Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis. Cambridge University Press. https://doi.org/10.1017/CBO9781139004244
- The Ethnographic Explorer, which generates the ranking data this tool is designed to analyse, is described in an earlier post: Making implicit knowledge explicit
- EvalC3 — a related tool that uses configurational logic to classify cases against user-defined outcome criteria, with a different interface and methodology.
