Asking Clinicians About Patients, Not Trees: Prior Predictive Elicitation over Rashomon Sets

Interpretable clinical risk models carry an awkward secret: there are usually thousands of them. Recent tree-learning algorithms can list every decision tree that fits the data almost as well as the best one. This list is called the Rashomon set. On some benchmarks it holds more than a million trees, and its members can disagree sharply about which patients are high risk and why. Today a clinician is handed this set and asked to pick a tree by eye. We propose a different interaction. The clinician is asked a few concrete questions about patient groups, for example "of ICU patients over 80 with low blood pH, what fraction do you expect to die in hospital?" The answers are then turned, by a principled statistical procedure, into a probability weighting over the trees in the set. Each question is chosen to fall exactly where the near-equally-accurate trees disagree most, so a handful of answers can collapse a huge set onto the few trees that are both accurate and clinically sensible.

The statistical machinery comes from Bayesian prior elicitation, the problem of turning expert knowledge into a prior distribution. That literature has learned that experts give poor answers about abstract model parameters and much better answers about observable outcomes. Its tool for this is the prior predictive distribution: the distribution of data a model predicts before seeing any data. So far that tool has only been used for smooth, low-dimensional parametric models. Our observation is that a Rashomon set is an ideal target for it. The set is a finite, discrete model space, so inverting expert answers into a prior needs no gradients through tree structure. Every model in the set already fits the data, so the elicited prior decides among accurate models instead of fighting the evidence.

Core Claim

Two open problems can each serve as the solution to the other. Rashomon-set methods produce many near-optimal interpretable models but give no principled way to choose among them. Prior predictive elicitation gives a principled way to turn observable judgments into priors, but its open problems include non-identifiability and the difficulty of differentiating through discrete structure. We claim that eliciting subgroup-level outcome rates, with questions picked actively where Rashomon members disagree, yields a posterior over interpretable clinical trees with three properties. First, it keeps the accuracy of the best tree. Second, it measurably reduces reliance on spurious, site-specific features. Third, it transfers better across hospitals than likelihood-only selection or hard rule-based filtering. All of this should take on the order of ten expert questions.

Relation to Prior Work

The method comes from two elicitation papers. Flexible Prior Elicitation via the Prior Predictive Distribution treats an expert's probabilities over binned observable outcomes as draws from a Dirichlet distribution centered on the model's prior predictive. It fits hyperparameters by maximum likelihood and estimates a precision parameter $\alpha$ that measures how consistent the expert is. It showed that direct parameter elicitation in a growth model gave "wildly inaccurate results" while predictive elicitation recovered reference values. Simulation-based prior knowledge elicitation for parametric Bayesian models does the same by matching simulated and elicited statistics under maximum mean discrepancy. It needs a Gumbel-Softmax relaxation to keep discrete likelihoods differentiable. We reuse the Dirichlet link from the first paper and the discrepancy matching from the second, the latter as an ablation. Both papers work in continuous parameter spaces. We change the support to a finite set of trees, where the inversion becomes an optimization over a simplex and the structural-differentiability problem they flag disappears.

The model spaces come from Falling Trees: A Model Class for Interpretable Risk Prioritization and From Rashomon Theory to PRAXIS: Efficient Decision Tree Rashomon Sets. The first constrains trees so that predicted risk falls along every root-to-leaf path, which suits triage. It reports MIMIC-III ICU mortality accuracy near XGBoost. The second enumerates sparse-tree Rashomon sets at scale, with more than a million trees on Churn. Both are purely loss-driven and leave the final choice to unguided inspection. They motivate our proposal by documenting the multiplicity, and they constrain it: we must not lose their accuracy. Doctor Rashomon and the UNIVERSE of Madness uses Rashomon sets to bound variable importance under unobserved confounding. It shows that the set's disagreement carries information, but it adds no domain prior.

The closest conceptual neighbor is Human-in-the-Loop Interpretability Prior. It also places a prior over models learned from human responses. That prior, however, measures how quickly a person can simulate a model, through response time. It says nothing about whether the model's predictions are medically plausible. A two-node tree keyed on admission hour is fast to simulate and clinically absurd. Our prior targets exactly what theirs ignores, and we treat simplicity as a separate, already-handled constraint built into the Rashomon set. Exploring the Gap between Informal Mental and Formal Statistical Models and Prior Knowledge Elicitation: The Past, Present, and Future motivate the choice of question format: experts reason in event rates for concrete patient prototypes. AutoElicit uses language models as stand-in experts for priors on predictive models. We use it only as one simulated-expert source, not as the method.

Research Program

Let $\mathcal{R} = \{T_1, \dots, T_K\}$ be a Rashomon set built on a construction split of the training data. We place a prior $w$ on the probability simplex over $\mathcal{R}$. A query is a clinically described subgroup $S_j$, for example a conjunction of two or three binarized features. Each tree $T_k$ assigns $S_j$ a predicted outcome rate $r_k(S_j)$. We partition rates into bins $b = 1, \dots, B$. The prior predictive probability of bin $b$ for subgroup $S_j$ is then

$$p_{jb}(w) = \sum_{k=1}^{K} w_k \, \mathbf{1}[r_k(S_j) \in b].$$

The expert answers with a probability vector $q_j$ over the same bins, which is easy to collect with a chip-allocation interface. Following the Dirichlet link, we model $q_j \sim \mathrm{Dir}(\alpha\, p_j(w))$ and fit $w$ and $\alpha$ by maximizing $\sum_j \log \mathrm{Dir}(q_j; \alpha p_j(w))$. A light entropy penalty keeps $w$ from collapsing onto trees that are only weakly identified.

The final posterior is $\pi(T_k \mid D) \propto w_k\, p(D_{\text{val}} \mid T_k)$. It uses a separate validation split so that the data are not counted twice: once in building $\mathcal{R}$ and again in the likelihood. A low fitted $\alpha$ gives an explicit warning that the expert's beliefs are not representable by any accurate tree. That is a useful finding in its own right, because it flags either a model-class gap or a miscalibrated belief.

Queries are chosen actively. At each step we ask about the subgroup whose predicted-bin distribution under the current weights has the highest entropy. This is the subgroup where accurate trees disagree most, so the answer moves the most mass. Candidate subgroups are restricted to those with at least a minimum number of patients and to features clinicians can read. Several experts are handled by fitting one shared $w$ with a separate $\alpha$ for each expert, so inconsistent experts are automatically down-weighted. This addresses the multi-expert disagreement problem that both elicitation papers leave open.

We will build this on Falling Tree Rashomon sets from GRAVITree and on sparse-tree sets from PRAXIS. For sets with millions of members, PRAXIS's enumeration lets us evaluate $r_k(S_j)$ in a batch. Where the set is too large to hold weights for every tree, we optimize over clusters of trees with identical prediction vectors on the candidate subgroups, which is lossless for this objective.

Baselines

The naive reference point is selection by data alone. We place a uniform prior over the same Rashomon set and pick by validation likelihood, which in practice means choosing the best training tree. On MIMIC-III ICU mortality, Falling Trees reports test accuracy of about 0.71 for this kind of selection and 0.73 for XGBoost on the same binarized data. Our method must stay within the Rashomon tolerance of 0.71 on in-distribution test data, which is the price we accept. It must beat that baseline on cross-site transfer and on spurious-feature reliance. All arms share the same binarization, Rashomon tolerance, and splits.

The strongest existing competitor for the selection problem is hard constraint filtering. The expert states each answer as a hard interval ("this subgroup's risk is between 20% and 40%"), and we keep only the trees that satisfy every interval. It uses the same questions, asked in the same order chosen by the same active rule, so the only difference is soft Dirichlet weighting versus hard inclusion. Beating it means better cross-site AUC or accuracy when expert answers are noisy, and no empty-set failures. Under perfectly reliable experts we expect a tie, and we will report it honestly.

The strongest unconstrained predictor is XGBoost tuned with the same hyperparameter budget on the same features. We pair it with SHAP explanations to measure attribution to the injected spurious feature. We do not expect to beat its in-distribution accuracy. We do expect to beat its reliance on the spurious feature, and we aim to match or beat it under cross-site shift.

Three ablations isolate the key ingredients. Random query selection, against disagreement-driven selection at equal question counts, isolates the active choice of subgroup. Replacing the Dirichlet likelihood with maximum-mean-discrepancy matching in the style of Simulation-based prior knowledge elicitation isolates the likelihood form and the value of $\alpha$. Applying the same elicited weights to an arbitrary Bayesian prior over all trees, rather than only the Rashomon set, tests whether restricting the support matters.

Why not just…?

Why not just let clinicians inspect the Rashomon set and pick a tree? This is what GRAVITree currently leaves to the user. It does not scale: PRAXIS reports more than a million competitive trees on Churn. Even a short list of a few dozen trees pushes clinicians toward judging how familiar a tree's structure looks rather than how plausible its predictions are. In a small clinician study we will compare picking from a list of 20 trees against our questions, measuring time and the cross-site performance of the chosen tree.

Why not just filter with hard rules? This is the real threat, and it appears above as a baseline run under the same conditions. Hard filters work when experts are precise and correct. They fail silently when one confident but wrong answer removes the best trees, or when the set empties. We settle this by sweeping the noise level in the simulated experts. If hard filtering matches soft weighting at realistic noise levels, the contribution reduces to active query selection, and we will say so.

Why not just train a black box and explain it with SHAP? Interpreting Sepsis Prediction through Positional Explanation shows that standard attribution can rank Gender as a top sepsis predictor purely because of where it sits in the input sequence. Post-hoc explanations describe a model; they do not change it. Our spurious-feature experiment measures directly whether the black box relies on the artifact while elicited trees avoid it.

Why not just ask clinicians for feature importances or coefficient ranges? Direct elicitation in parameter space gave "wildly inaccurate results" in Flexible Prior Elicitation via the Prior Predictive Distribution. A tree also has no coefficients to elicit. We include a feature-ranking prior, which penalizes trees that use features the expert ranks low, as an extra arm with the same expert source.

Proposed Experiments

The first testbed is MIMIC-III (MIMIC-III, a freely accessible critical care database) ICU mortality, using the Falling Trees binarization. Into the training data we inject a synthetic shortcut: a binary feature correlated with mortality in training but independent of it at test time, mimicking a site documentation artifact. The question is whether a few elicited answers demote trees that use the shortcut.

The second testbed is PhysioNet 2019 sepsis (Early Prediction of Sepsis from Clinical Data), reduced to patient-level onset prediction from binarized summaries of the 40 variables. The natural shift here is between hospitals. We build Rashomon sets on Hospital A. The simulated expert's answers are subgroup rates from Hospital B, corrupted with calibrated noise. Evaluation is on held-out data from the other site. The challenge reports winning utility scores of 0.33 to 0.36. That figure applies to hourly sequence models, so we report it only as context and compare primarily by AUC under shift.

Expert sources come in three tiers. The first is oracle-plus-noise experts drawn from the other site, at controlled noise and bias levels. The second is language-model experts prompted as in AutoElicit. The third is a small study with critical-care clinicians who answer about ten subgroup questions each.

Metrics cover four things: test and cross-site AUC and accuracy; the posterior mass placed on trees that use the shortcut feature; the number of questions needed to reach 90% of the final cross-site gain; and the fitted $\alpha$ for each expert as a consistency diagnostic.

The decisive checks are four. First, elicitation with ten or fewer actively chosen questions must cut posterior mass on shortcut-using trees by a large margin relative to likelihood-only selection, without falling below the Rashomon accuracy tolerance. Second, cross-site AUC must improve over both likelihood-only selection and hard filtering once expert noise reaches realistic levels. Third, active selection must need clearly fewer questions than random selection. Fourth, $\alpha$ must track the injected expert noise, so that unreliable or adversarial experts are down-weighted. If the first check fails, meaning experts' subgroup rates cannot tell shortcut trees apart, then observable elicitation is the wrong lever for Rashomon selection, and that negative result is worth reporting.