Where Ensembles Buy Tail Safety: A Decomposition of Ensemble CVaR

Averaging several independently trained networks is known to help most on the hardest inputs. Deep ensembles raise the accuracy of rare subgroups and the worst classes far more than they raise average accuracy. Nobody can yet say, in the vocabulary of risk measures, why this happens or how to get the benefit cheaply. We propose an exact identity that splits an ensemble's tail risk into three measurable parts. Tail risk here means Conditional Value-at-Risk (CVaR), the average loss over the worst α-fraction of examples. The three parts are the members' own tail risk, a term that measures how far the members' worst examples fail to overlap ("tail misalignment"), and a term that measures how much the members disagree exactly where the ensemble itself fails ("tail diversity"). Every term can be computed on held-out data for any pooling rule, including the power means that interpolate between probability averaging and logit averaging. We use the identity for two things. First, we diagnose which sources of member variation actually reduce tail risk. Second, we test a practical claim: pruning a large seed ensemble by validation CVaR keeps most of the tail benefit of 20 or more models with a handful of members.

This matters because the evidence for "ensembles are fair" is currently a heuristic finding about uniform averaging. FAIR-Ensemble trains identical networks that differ only in random seed. It finds that top-line accuracy saturates after 4–5 members, while worst-class and minority-subgroup accuracy keeps rising up to 20–50 members, and it attributes this to disagreement concentrated on hard samples. That is a strong result with a large price tag and no risk-theoretic account. The existing risk-aware ensembling work either randomizes predictions at inference or targets average likelihood, and neither answers the question.

Core Claim

Fix a pool of $M$ members and a pooling rule. For each labeled example, let $\ell_m$ be member $m$'s loss and $\ell_E$ the pooled ensemble's loss. Define the pointwise pooling gap $J = \frac{1}{M}\sum_m \ell_m - \ell_E$. This gap is nonnegative whenever the loss is convex in the pooled prediction. Two examples are the squared-error ambiguity term and, for cross-entropy under probability averaging, a Jensen gap. Let $q^*$ be the weighting that attains $\text{CVaR}_\alpha(\ell_E)$; empirically, it is uniform weight on the worst α-fraction of examples under the ensemble. Then, exactly,

$$\text{CVaR}_\alpha(\ell_E) = \frac{1}{M}\sum_{m} \text{CVaR}_\alpha(\ell_m) \;-\; \frac{1}{M}\sum_m \Big[\text{CVaR}_\alpha(\ell_m) - \mathbb{E}_{q^*}\ell_m\Big] \;-\; \mathbb{E}_{q^*}[J].$$

The middle bracket is nonnegative because $q^*$ lies in CVaR's set of admissible reweightings. It is zero only when every member's worst examples coincide with the ensemble's worst examples, so we call it tail misalignment. The last term is tail diversity: disagreement measured only on the ensemble's own failures.

The identity follows from CVaR's dual representation, so it holds verbatim for any coherent risk measure with an attained maximizing weighting. That includes expectiles at levels above one half and distortion measures.

The identity itself is elementary. What is new is that it replaces the second-order Taylor approximation of Generalized Ambiguity Decomposition. That work decomposes the expected loss of smooth convex losses, and it breaks down when losses are non-smooth or when members are far from the target. Our identity is exact, and it measures diversity under the tail weighting rather than the data average.

It also turns the churn story of FAIR-Ensemble into a falsifiable prediction. We expect the continued tail gains between 5 and 50 members to show up as growing misalignment and tail diversity while average member loss stays flat.

For power-mean pooling, Beyond Mixtures and Products proves that orders $r\in[0,1]$ are the safe zone for average log-likelihood, and it reports an NLL-optimal $r\approx 1.4$ on CIFAR-100. In our identity, $J$ can turn negative on individual examples outside that range. The identity therefore lets us ask whether the tail-optimal order differs from the likelihood-optimal one, and whether the difference is explained by negative $J$ on tail examples.

We must address the "deterministic trap" of Boosted CVaR Classification directly. That paper proves that for deterministic classifiers under 0/1 loss, minimizing CVaR and minimizing average error pick the same model, which is why it randomizes over ensemble members at inference. A deterministic pooled ensemble that outputs an argmax is still a deterministic classifier. Our framing therefore does not claim to escape that trap under 0/1 loss. We measure CVaR of the continuous per-example loss (cross-entropy, or squared error in regression) and report group-level worst-case accuracy separately. This is the regime where deterministic pooling can matter, and it is what regulated deployments that forbid non-repeatable predictions actually need.

Research Program

The first component is a measurement study. On each benchmark we train a pool of 50 homogeneous ResNet members, following the FAIR-Ensemble protocol. We vary four sources of variation separately: initialization only, mini-batch order only (which FAIR-Ensemble found to be the stronger driver of diversity), bootstrap resampling, and small-subsample training as in Subsampled Ensemble Can Improve Generalization Tail Exponentially. We add a fifth pool whose members are each trained with a CVaR objective at a different level α. This borrows the multi-level idea of RM-CVaR, which hedges CVaR's sensitivity to the risk level across several levels. RM-CVaR applies that idea to asset weights; we apply it to predictive models.

For each pool, each ensemble size, and each pooling order $r\in\{-1,0,0.5,1,1.4,2\}$, we compute the three terms at α ∈ {0.01, 0.05, 0.1}. We attach bootstrap confidence intervals, because tail estimates converge slowly; the RUCC analysis reports an $O(1/\sqrt{T})$ rate for CVaR control (RUCC).

The second component is a cheap construction method. We greedily add members from the pool to minimize CVaR on a held-out split, and in a second variant we also select the pooling order $r$. We then report how few members recover the 50-member tail risk. We deliberately use direct validation-CVaR selection rather than a decomposition-derived score, because the direct criterion is the stronger method. The decomposition's role is to explain which members greedy selection picks: whether it favors misaligned members or members that disagree on the tail.

The third component extends the analysis to regression. We use UCI regression tasks with injected Pareto noise, where squared loss makes $J$ exactly the classical ambiguity term. This lets us test whether mode-based voting (ROVE), which Subsampled Ensemble shows turns the generalization tail from polynomial to exponential, also lowers the CVaR of per-example loss. The alternative is that ROVE improves the tail over training draws while leaving the tail over test examples untouched. These are two distinct notions of "tail" that the literature currently conflates.

Baselines

The naive reference points are a single ResNet-18 trained with standard empirical risk minimization, and a single wider network with compute matched to the ensemble. On the CelebA Blonde-Male subgroup, FAIR-Ensemble reports 9.44% accuracy for the single ResNet-18.

The strongest deterministic competitor is the uniform 20-member seed ensemble of FAIR-Ensemble, which reaches 21.80% on that subgroup. We rerun it on identical splits, architectures, and augmentation. Beating it means one of two outcomes. The first is matching its subgroup accuracy and test CVaR within confidence intervals using at most 5 members. The second is exceeding it at equal member count.

The strongest risk-aware competitor is α-AdaLPBoost from Boosted CVaR Classification. It computes CVaR-optimal mixture weights over fixed base models and samples one member per query. The file reports no numbers on COMPAS, CelebA, or CIFAR ("no reported result found on Lacuna for Boosted CVaR on these benchmarks"), so we will run it on our exact pools with the same validation data. We compare it on both randomized expected loss and on a deterministic variant that pools with its weights. The fair test is whether deterministic greedy pooling comes within its CVaR at equal member budget.

We also include power-mean pooling with the NLL-tuned order from Beyond Mixtures and Products; its reported optimum is $r\approx 1.4$ on CIFAR-100. For regression we include bagging and ROVE.

The key ablation replaces the validation-CVaR selection criterion with validation-mean-loss selection, using the same greedy procedure and budget. If the two pick the same members, tail-targeted construction adds nothing.

Why not just…?

Why not just train one bigger network? The decomposition makes this concrete: a single model has no misalignment or tail-diversity terms, so any tail gain must come from lowering its own CVaR. The compute-matched wide network in our baselines tests whether capacity alone does that. The FAIR-Ensemble contrast between 9.44% and 21.80% suggests it will not, but we measure it rather than assume it.

Why not just average all 50 seeds uniformly? Uniform averaging is a genuine competitor on tail quality, and it may simply win. The case against it is cost: FAIR-Ensemble itself needs 20–50 networks. The pruning experiment settles whether that cost is avoidable. If five greedily chosen members fall well short of fifty uniform ones, we report it as a negative result about compression.

Why not just greedily select members by validation CVaR, with no decomposition? This is a real threat, and that is why we adopted it as the method rather than arguing against it. The decomposition does not claim to select better. It claims to explain what is selected and which training intervention to prefer: seed type, subsampling, or multi-level CVaR members. That claim is tested by checking whether the terms predict held-out tail gains across pools better than average-loss diversity does.

Why not just use randomized CVaR boosting? Randomized prediction is unacceptable wherever identical inputs must receive identical outputs. Boosted CVaR's own analysis leaves open whether deterministic pooling can approach its tail performance. Our head-to-head comparison at equal budget answers that directly.

Decisive Checks

The work succeeds or fails on four results. First, on CelebA, TinyImageNet's bottom-10 classes, DermaMNIST, and COMPAS/Adult, the misalignment and tail-diversity terms must grow with ensemble size past the point where average loss saturates. If they stay flat, the churn explanation of FAIR-Ensemble is wrong in risk terms. Second, the mini-batch-order pool should show larger tail terms than the initialization-only pool, consistent with FAIR-Ensemble's diversity finding. Third, at five members, CVaR-greedy pruning should recover at least most of the 20-member subgroup gain, measured by accuracy on the CelebA Blonde-Male subgroup and by test CVaR at α=0.05. Fourth, we check whether the CVaR-optimal pooling order $r$ departs from the NLL-optimal one on CIFAR-100. If it does, we check whether the shift coincides with negative pooling gaps on tail examples, which would identify a concrete way likelihood-tuned aggregation misprices tail risk.