When rankings tell different stories: an introduction to the Rank Explorer

 

Caveat Emptor: I delegated the writing of this blog posting to Claude AI (Sonnet 4.6), based on an extended prior dialogue on the subject below, then a summary prompt of what was wanted. My post-production edits were quite limited.

--o0o--

There is a situation that turns up repeatedly in evaluation and research practice that is easy to overlook precisely because it looks like a data analysis problem rather than a methodological one. The situation is this: you have a set of cases that have been ranked on multiple factors, along with a ranking of their outcomes, and you want to understand what the relationship is between the factors and the outcome.

This sounds straightforward. And to a degree it is — you can correlate each factor against the outcome, identify the strongest relationships, and build a composite ranking that aggregates them. These are useful things to do. But they share a common assumption that is not always warranted: that the factors combine additively, that each one contributes independently to the outcome, and that a case that scores well on several factors will therefore tend to score well overall.

That assumption is often wrong. And when it is wrong, the gap between your composite ranking and the actual outcomes is telling you something important.

The problem with additive aggregation

Consider a concrete example. You have 63 local authority areas. You have ranked them on ten factors thought to be associated with population-level physical activity — access to green space, deprivation levels, sports facility density, and others. You have also ranked them on an outcome measure. You build a composite ranking from the factors, correlate it with the outcome, and find it predicts reasonably well — perhaps a Spearman r of 0.75.

That is a decent result. But it is hiding something. Some areas with strong factor rankings are performing poorly on the outcome; others with weak factor rankings are performing well. If you look closely, there are two or three quite different combinations of factors that each seem sufficient, on their own, to predict a good outcome. These are not variations on the same story — they are distinct causal pathways. However an additive composite averages across them, and in doing so obscures the structure.

This is what researchers in the QCA tradition call equifinality — multiple routes to the same outcome. Additive methods cannot find it. A decision tree can.

What the Rank Explorer does

The Rank Explorer is a browser-based companion tool to The Ethnographic Explorer. It is designed to import the rankings data that TEE generates from its Contrast tab, though it will also accept ranking data from any other source in the same CSV format.

[▶ Try the Rank Explorer] 

The tool has four analysis tabs. The first three — Individual Factors, Composite Builder, and Scatter Plot — provide standard additive analysis: Spearman correlations for each factor, composite rankings using several aggregation methods (equal-weight, correlation-weighted, stepwise greedy, and exhaustive search), and a scatter plot that visualises how well your composite ranking predicts the outcome, with adjustable classification thresholds.

The fourth tab, Pathway Explorer, is where the configurational logic comes in. It builds an optimal classification tree over your data using exhaustive search: at each node, every available factor and every possible rank cut-off is tested, and the split that best separates high-outcome from low-outcome cases is chosen. The result is a tree that shows which specific combinations of factor ranks distinguish the two groups, displayed in a row-by-level icicle layout that makes the branching structure easy to follow.

The tree is not just a visual. Each leaf node shows which cases ended up there, whether they were correctly classified, and the conditions that led to that grouping. A pathway summary below the tree lists the conditions for each leaf in plain language — for instance, "Active travel infrastructure: rank 5 or better AND Deprivation index: rank 8 or better."

When the gap between additive and configurational results is itself a finding

One of the more useful diagnostics the tool enables is comparing the classification accuracy of the best composite ranking against that of the decision tree. If both are similar, the additive story is probably adequate. If the tree substantially outperforms the composite — reaching, say, 90% or 100% accuracy where the composite only reached 75% — that gap is a finding in itself: the causal structure in the data is better described by conjunctions of conditions than by sums of contributions.

This matters for intervention design. If a high-outcome classification requires both good green space access and good active travel infrastructure (rather than either being substitutable for the other), then improving one without the other may produce no discernible effect. Additive analysis will not surface that conclusion; configurational analysis will.

A note on scale, depth, and selective deepening

The tree-building algorithm uses exhaustive search, which is thorough but computationally intensive. With datasets of 60–70 cases and 10 factors, a depth-3 tree typically builds in a few seconds. Depth 4 or beyond is a different matter: computation time increases steeply, and more importantly, deeper trees on small datasets will often find spurious distinctions — patterns that reflect the quirks of the sample rather than anything real.

The Rank Explorer addresses this through a subgroup analysis feature that may be a modest innovation in decision tree practice for small-N datasets. Once a tree has been built, each leaf node displays an Analyse subgroup → button. Clicking it filters the dataset to only the cases in that leaf and opens a fresh analysis session for that group alone — with all four tabs, including the Pathway Explorer, reconfigured for the subgroup. The outcome cut-off resets automatically to the median of the subgroup's outcome ranks, so the high/low distinction remains balanced within the smaller group.

This allows selective deepening of a specific branch without rebuilding the entire tree at greater depth. If one leaf contains 24 cases that the main tree could not separate further, the subgroup analysis asks a different and analytically legitimate question: within this group, what distinguishes the relatively better-performing cases from the worse ones? The answer applies conditionally — only to cases that reached that leaf — but that conditionality is precisely what makes it interpretable. A banner remains visible throughout the subgroup session as a reminder that the high/low labels are relative to the subgroup, not the full dataset.

For larger or more complex datasets, the stepwise greedy method in the Composite Builder tab is a useful preliminary step: it adds factors to the composite one at a time, selecting whichever remaining factor most improves the correlation with the outcome at each step. The resulting path table shows the marginal contribution of each factor, making it straightforward to identify a smaller subset that carries most of the predictive weight — before running the Pathway Explorer on that reduced set.

Beyond TEE data

The tool is designed as a TEE companion but is not restricted to it. Any CSV with a column of case names and a set of ranking columns will load correctly. Evaluation practitioners who have generated case rankings through other means — expert scoring panels, secondary data, peer comparison exercises — can use the same analytical workflow.

Some framings that could map onto the same tool:

• Programme portfolios: rank a set of projects on design-quality dimensions and an outcome measure, then identify which combinations of design features distinguish the most successful from the rest
• Organisational assessments: rank a set of partner organisations on capability dimensions, use the tree to find which combinations are most predictive of delivery performance
• Cross-country comparison: rank a set of countries or regions on contextual factors alongside a development indicator, and look for the configurational patterns that additive index approaches miss

In each case the structure is the same: cases, factor rankings, an outcome ranking, and the question of what the relationship looks like once you stop assuming it is additive.

An invitation to experiment

The tool is best explored with data you already have. If you have ever built a composite index and felt that it was not quite capturing something you could see in the data, or have had the experience of an outlier case that your model consistently misclassifies, the Pathway Explorer is a reasonable next step. Loading your own data, building a tree at depth 2 or 3, and comparing the pathway classification against your composite should take no more than a few minutes.

I am continuing to develop both tools and would welcome feedback on the approach, the interface, or uses I have not considered.

Accessing the code: The Rank Explorer runs entirely in your browser — no login required, no data is transmitted anywhere. To save your own copy of the code, open the tool, right-click, select View Page Source, copy the entire code, paste it into a text file, rename it to end in .html rather than .txt, and open it in any web browser.

Further reading

• Schneider, C. Q., & Wagemann, C. (2012). Set-Theoretic Methods for the Social Sciences: A Guide to Qualitative Comparative Analysis. Cambridge University Press. https://doi.org/10.1017/CBO9781139004244 
• The Ethnographic Explorer, which generates the ranking data this tool is designed to analyse, is described in an earlier post: Making implicit knowledge explicit
• EvalC3 — a related tool that uses configurational logic to classify cases against user-defined outcome criteria, with a different interface and methodology.

Introducing Rank Order Counterfactuals (ROC)

A counterfactual is a description of what would have happened, if an intervention had not taken place. The use of randomised control groups is one way to construct a counterfactual. A population of people are randomly assigned to either a control group or an intervention group. Differences in the outcomes of those populations are then compared. If the difference is sufficiently statistically significant then a plausible causal claim can be made that difference in outcomes is because of the intervention.

As might be expected, there are plenty of circumstances when social programs are designed and implemented, where it is simply not practical to organise a randomised control group. In addition, the comparisons that are made will be between averages of the two groups. However, in many social programmes such averages are of limited practical use, because the implementation contexts are so varied and no single “solution” is likely to be applicable. Average effects can still be informative at a high level, but they need to be complemented by methods that take contextual diversity seriously.

I'm currently working with an evaluation team that is examining a large-scale public health programme in the United Kingdom, covering many different locations and involving many different types of local partnerships. But with one common outcome of concern, which is to increase people's physical activity levels in their daily life. In their work the evaluation team is already making use of a causal configurational approach to the understanding of what works for whom in what circumstances. It is finding different configurations of causal conditions across these locations that are associated with changes in activity levels. This approach is consistent with the high level of diversity in locations partnerships and interventions.

But what it does not yet have is a counterfactual, a defensible description of what might have happened in these locations in the absence of this intervention. This is where the idea of a rank order counterfactual becomes relevant. By a rank‑order counterfactual I mean a very specific kind of “what would have happened otherwise.” Instead of trying to predict the exact outcome that would have been achieved in each location without an intervention, we can start by asking a simpler, comparative question: which location would probably have changed more, and which less, if the intervention had never existed? The answer will be in the form of a rank ordering of locations, from those with more to less expected change. That ranking would be constructed based on all available baseline information, trends, and contextual knowledge. This proposed approach falls into a category of counterfactuals known as "logically constructed counterfactuals", and it aligns well with configurational evaluation because it focuses on patterns of relative change across diverse contexts.

A subsequent evaluation of those same locations should also be able to generate a new rank ordering, which is based on observed outcomes. These counterfactual and actual rankings can then be compared, using a scatterplot and correlation measures. The scatterplot is also visually powerful for communication: it lets people see at a glance which locations behave as expected and which ones stand out as surprises. If the intervention had no effect we should see a linear relationship, the observed and counterfactual rankings should be the same. If the intervention had positive, or perhaps even negative effects, this should not happen. We might see various locations which are outliers from that expected trend. When locations we expected to be “natural leaders” did not improve much, and those we expected to be “natural laggards” moved to the top of the league table, that pattern is a signal that the intervention may have been influential, and it gives the evaluation team clear cases where alternative explanations should be probed. The task of the evaluation is then to probe those alternative explanations, not to assume the intervention is the only possible cause. The rankings are not a substitute for theory‑based evaluation; they are a way to make its claims sharper and more testable. The focus on ranking differences can convert a vague theory (“we think these factors matter”) into a concrete, specific prediction about which locations should do better.

The sensitivity of the rank comparison process will depend on the number of ranked items. The more rank positions there are, the more sensitivity there will be to differences in performance, which is good. But, as shown in research on sorting algorithms, the time required to generate a complete sorting, using any of the well-known methods, can be significant. Growing faster than proportionally to the number of items, though far slower than exponential growth. In addition to the extra time required, a rank order counterfactual will require a stronger evidential base where the number of rank positions is greater. 

When a large number of locations are involved in an intervention one practical way of addressing this tension is to use a stratified random sample, and to generate the rankings for that sample only. Another approach to managing large numbers of locations is to think of ranked bands of locations rather than individual rankings for each location. What should be of interest, then, are systematic shifts in band membership between the counterfactual and actual observations – for example, locations expected to be in the “low‑change” band turning up in the “high‑change” band in practice.

In this way, rank‑order counterfactuals do not replace theory‑based evaluation, but sharpen it: they turn general expectations about context into explicit, testable predictions about who should have changed most in the absence of the programme. In work which I hope to document in the European Evaluation Society conference later this year I will explain how the use of the hierarchical card sorting process was used to generate argument and evidence based counterfactual rank orderings, and how an LLM was used to support this process.