How Much Should We Trust a Narrowed Climate Projection?

Climate projections come from dozens of physics simulators that disagree even when fed identical emissions. A popular way to narrow that disagreement is the "emergent constraint". Across the ensemble of models, one finds a present-day quantity that we can observe, such as a historical warming trend, and that correlates with the future quantity we care about. One then reads the future off the real-world observation. This note argues that the procedure is really an observational study with two unexamined weaknesses. First, many different constraints fit the ensemble almost equally well yet imply different futures; the ML fairness literature calls this predictive multiplicity. Second, models that share code form families, and that shared heritage is a hidden common cause of both the observable and the projection; causal inference calls this unobserved confounding.

We propose to report narrowed projections as intervals that stay honest about both weaknesses. We also propose a single interpretable number: how much hidden family-level confounding would be needed before the narrowing disappears. The approach reuses machinery from causal sensitivity analysis and from Rashomon-set analysis, and applies it to the ensemble itself. Each climate model is treated as one data point. The ensemble is treated as a biased, non-random sample from the space of plausible models.

Core Claim

The inter-model spread in an ensemble like CMIP6 can be analysed with the tools used for observational data, where each "unit" is a simulator. Seen this way, an emergent constraint is a regression fitted on about 36 correlated units. It suffers from the two problems that the other two research directions study.

The Rashomon set is the set of models whose error lies within a small tolerance of the best model's error (Predictive Multiplicity in Classification). Here it consists of candidate constraint regressions, each built on different observables, whose leave-one-model-out error is within a tolerance $\epsilon$ of the best. Members of this set can disagree sharply about the constrained future.

Shared code heritage acts like a selection mechanism. Some regions of model space are over-sampled because several institutions inherited the same parameterization. We can bound its effect in the way that marginal sensitivity models bound hidden confounding. Such a model limits how far the true sampling or treatment weights can deviate from the observed ones, by a factor $\Gamma \ge 1$.

The new object is the union, over the Rashomon set, of the $\Gamma$-bounded reweighted projections. It comes with a breakdown value $\Gamma^\*$: the smallest amount of hidden heritage confounding at which the constrained interval becomes as wide as the unconstrained ensemble range. We claim two things. The interval should achieve its nominal coverage in perfect-model tests where ordinary emergent constraints do not. And $\Gamma^\*$ should separate constraints that survive scrutiny from those that are statistical accidents.

Concretely, let model $i$ have observable $X_i$, projection $Y_i$, and nominal weight $w_i$ (equal weights, or family-aware weights). The sensitivity-adjusted upper bound for a constraint $h$ from the Rashomon set $\mathcal{R}(\epsilon)$ is

$$\overline{Y}_\Gamma(h) = \max_{\tilde w:\ w_i/\Gamma \le \tilde w_i \le \Gamma w_i} \frac{\sum_i \tilde w_i\, \hat Y_i(h, X^{obs})}{\sum_i \tilde w_i},$$

where $\hat Y_i(h, X^{obs})$ is the projection that constraint $h$, applied to the real-world observation $X^{obs}$, attributes to model $i$ once model $i$'s residual is included. The lower bound $\underline{Y}_\Gamma(h)$ is the same expression with a minimum in place of the maximum. The reported interval is $[\min_{h \in \mathcal{R}(\epsilon)} \underline{Y}_\Gamma(h),\ \max_{h \in \mathcal{R}(\epsilon)} \overline{Y}_\Gamma(h)]$. Like marginal-sensitivity bounds, each inner optimisation is solved exactly by sorting the models and thresholding the weights. This keeps the method cheap even when the Rashomon set is enumerated exhaustively.

Relation to Prior Work

Two lines of work motivate the gap.

Opinion: Why all emergent constraints are wrong but some are useful – a machine learning perspective argues that cross-ensemble relationships may be spurious rather than physical. It offers that argument qualitatively; we turn it into a measurable quantity, $\Gamma^\*$.

Constraining uncertainty in projected precipitation over land with causal discovery weights CMIP6 models using two ingredients. One is how well each model's causal network over 60 sea-level-pressure modes matches reanalysis. The other is a penalty for redundancy between models. The resulting weights shrink projected precipitation ranges. That work assumes no hidden confounders among its modes and commits to one weighting. We keep weighting as an input, but ask how far its conclusion survives when the weights themselves could be off by $\Gamma$.

Two papers supply the methods.

Auditing Fairness under Unobserved Confounding bounds a treatment-allocation disparity under a confounding parameter $\gamma$. It reports robustness up to $\gamma = 1.12$ and calibrates that number by saying a hidden factor would need to be "stronger than the effect of diabetes." We borrow this calibration style. The analogue here is to say that hidden heritage would need to be stronger than the known effect of belonging to a documented model family.

Doctor Rashomon and the UNIVERSE of Madness is the closest existing combination of Rashomon sets and unobserved confounding. It widens Rashomon sets to account for finite samples and omitted variables, and it bounds variable importance in classification. We apply the same two-layer logic to a different target: a real-valued scientific projection rather than a feature importance. The sample is also a different kind of thing, roughly 36 correlated simulators rather than thousands of individuals. We reuse its finite-sample expansion of $\epsilon$, which matters even more at this sample size.

Two papers constrain the design.

Reimagining Meaningful Model Multiplicity argues that multiplicity is only meaningful if no single predictor could dominate the alternatives. We therefore test whether the disagreement within our Rashomon set is concentrated in particular regions. If a single regional multiaccurate predictor removed that disagreement, our interval would be unnecessary.

ClimateSet shows that emulators tuned on one climate model transfer poorly to others. Our cross-model regressions face the same problem, which is why every evaluation below holds out whole models or whole families.

Research Program

We first build the unit-level dataset. We use the CMIP6 models in ClimateSet, with their historical and SSP outputs for surface temperature and precipitation. We add model-genealogy labels that group models sharing atmospheric or land components into families, as in the interdependence concerns raised by Debeire et al.

Candidate observables are historical regional warming trends, precipitation seasonality, and the global mean trend. Target projections are end-of-century (2081–2100) changes in global and regional temperature and land precipitation under SSP2-4.5, SSP3-7.0 and SSP5-8.5.

The constraint hypothesis class has two tiers. The first is sparse linear regressions over subsets of observables; these can be enumerated exactly. The second is small ridge and gradient-boosted maps from historical fields to projected change. For the second tier we explore the Rashomon set by retraining across seeds and hyperparameters, following The Role of Hyperparameters in Predictive Multiplicity.

We then compute three things. The first is ambiguity: how far apart equally good constraints place the real-world projection. The second is the $\Gamma$-bounded intervals. The third is $\Gamma^\*$ for each target.

Finally, we answer a question the file flags as open. Is the multiplicity among equally good constraints within the ensemble larger or smaller than the inter-model spread it is supposed to reduce? If it is comparable, a single emergent constraint misrepresents what the ensemble actually supports.

Baselines

The naive reference point is model democracy. It reports the unweighted 5th–95th percentile range of the ensemble, with no constraint, and every narrowing claim is measured against it.

The standard emergent constraint is the second baseline. It is one ordinary-least-squares regression on the best-correlated observable, with its prediction interval evaluated at the observation. This is what most published constraints do, and it is the method our Rashomon layer directly criticises.

The strongest existing competitor is causal-discovery weighting as in Debeire et al. It reports that very likely ranges of land precipitation change shrink by between 10% and 16%, and likely ranges by between 16% and 41%, across SSP2-4.5, SSP3-7.0 and SSP5-8.5. We reimplement it on the same models, using CMIP6 sea-level pressure, which we obtain for the ClimateSet model list. We use the same reanalysis references and the same target periods. Our method then gets the same inputs and uses Debeire weights as its nominal $w_i$. That makes the comparison purely about whether the added robustness layer buys calibration.

A family-aware independence weighting is the fourth baseline. It down-weights models by family size, with no causal fingerprints. We include it because it is the cheapest direct fix for heritage.

Two ablations isolate the key ingredients. Setting $\Gamma = 1$ gives multiplicity without confounding. Using only the single best constraint with $\Gamma > 1$ gives confounding without multiplicity.

All methods share the same models, observables, targets and hold-out splits. Wherever a method has free choices, such as $\epsilon$, $\Gamma$, or the observable set, we tune it with the same nested leave-one-family-out budget.

Beating the baselines means the following. Our interval reaches nominal 90% coverage in perfect-model tests where the standard constraint and Debeire weighting fall below it. At the same time it stays measurably narrower than model democracy. On perfect-model coverage for these baselines, no reported result was found on Lacuna, so we will measure all of them ourselves.

Why not just…?

Why not just use equal-weight averaging or report the full ensemble range? This is always safe in the sense of coverage, but it throws away whatever real information the observations carry. It also over-counts duplicated model families, which is the central critique in Debeire et al. Our method includes this range as its $\Gamma \to \infty$ limit, so the question is purely whether a finite, defensible $\Gamma$ yields useful narrowing. The perfect-model coverage-versus-width curve answers that directly.

Why not just use family-aware independence weighting and stop there? This is the most serious threat. If down-weighting by known genealogy already delivers calibrated coverage in leave-one-family-out tests, then hidden heritage beyond the documented families is negligible. In that case the sensitivity layer is unnecessary, and we will say so. The decisive check is whether coverage failures persist after family weighting. If they do, the undocumented part of heritage matters, and $\Gamma$ is estimating something real.

Why not just average over many emergent constraints, for example with Bayesian model averaging? Averaging collapses the Rashomon set into a single mean. It reports a narrow interval precisely when constraints disagree in offsetting directions. The anchoring identity in Model Agreement via Anchoring shows that averaging two models lowers the average squared error by exactly the amount they disagree. Averaging therefore hides the disagreement, which is the quantity we want to surface. We will report the averaged constraint alongside ours. If its coverage matches ours at smaller width, the multiplicity layer is redundant for these targets.

Why not just impute the hidden heritage as a latent variable, as CI-StoNet does? Stochastic Neural Networks for Causal Inference with Missing Confounders imputes missing confounders with a stochastic latent layer. It reports average treatment effect errors of 0.0709 on ACIC 2019 and 0.0135 on Twins with gestational age hidden. But it needs a correctly specified causal graph and far more than 36 units. Neither condition holds for an ensemble of simulators. Bounding the hidden variable is more honest here than estimating it.

Proposed Experiments

Each experiment is a perfect-model test. In each fold, one model or one whole family is held out and treated as "truth". Its historical output plays the role of the observation, and we constrain its future using the remaining models.

The primary metric is empirical coverage of the nominal 90% interval, over folds and targets. We report it jointly with interval width relative to the unweighted range. Secondary metrics are the proportion of regions where members of the Rashomon set disagree by more than the unconstrained interquartile range, and the distribution of $\Gamma^\*$ across targets.

The first decisive check is whether standard emergent constraints and Debeire-style weighting under-cover when whole families are held out. The file motivates this suspicion but does not document it.

The second is whether our intervals restore coverage at a $\Gamma$ that is small relative to a calibration yardstick. The yardstick is the weight distortion implied by the known family structure, in the spirit of the diabetes comparison of Byun et al.

The third is whether $\Gamma^\*$ ranks constraints sensibly. Global temperature constraints, which have strong physical rationale, should survive larger $\Gamma$ than regional precipitation constraints. Chang et al. show that models reach similar precipitation projections through different physical mechanisms (Changes in Spatiotemporal Precipitation Patterns in Changing Climate Conditions), which is why precipitation constraints should be the more fragile.

If the first check fails, meaning ordinary constraints are already calibrated under family hold-out, the proposal's premise is wrong for CMIP6. In that case we would publish that finding instead.