Certifying How Many Independent Voters an Ensemble Really Has

An ensemble only helps when its models make different mistakes. Recent audits find that this assumption fails badly for modern models. Seventeen vision-language models behave like two to four independent voters. Six tabular foundation models have near-identical errors. Strong LLMs pick the same wrong answer far more often than chance would predict. Each of these conclusions rests on a correlation matrix estimated from a finite set of labeled questions, and none comes with a statement of how wrong that estimate might be. Practitioners then prune pools, drop "clones," and keep "diversity heroes" based on point estimates. Sometimes the sample is only 113 forecasting questions.

We propose to put finite-sample confidence guarantees on these redundancy judgments. The tool is a family of results from random matrix theory that we call intrinsic-dimension concentration bounds. These are tail bounds for averages of random matrices whose cost depends on how many directions the variance actually spreads over, not on the size of the matrix. That property fits ensembles well, because a pool that is highly redundant is exactly a pool whose error matrix has a small intrinsic dimension. The deliverable has three parts: a confidence region for the ensemble's error matrix, valid simultaneously for every question asked of it; certified lower and upper bounds on the effective number of independent voters; and a pruning rule that removes a model only when its redundancy is statistically certified. We will test whether these guarantees stay tight enough to change decisions on MMLU, HELM, VQA, and forecasting pools.

Core Claim

Let $x_j \in \{0,1\}^m$ record which of $m$ models get question $j$ wrong. The population error matrix is $S = \mathbb{E}[x x^\top]$, and its estimate from $n$ labeled questions is $\hat S = \frac{1}{n}\sum_{j=1}^n x_j x_j^\top$. Its intrinsic dimension is $d_{\text{int}} = \mathrm{Tr}(S)/\lambda_{\max}(S)$, the total error mass divided by the largest error mass along any single direction. When models are clones, the error mass lies mostly along one direction and $d_{\text{int}}$ is small.

We claim that a confidence ball

$$\|\hat S - S\| \le \sqrt{\frac{2\hat\sigma^2 \log(c\,\hat d_{\text{int}}^{+}/\delta)}{n}} + \frac{R\log(c\,\hat d_{\text{int}}^{+}/\delta)}{3n}$$

can be computed from data alone and holds with probability $1-\delta$. In this bound, $\|\cdot\|$ is the spectral norm, $\hat\sigma^2$ is an empirical upper bound on the matrix variance, $R$ bounds $\|x_j\|^2$, $\hat d_{\text{int}}^{+}$ is an upper confidence bound on $d_{\text{int}}$, and $c$ is a small absolute constant. The Bernstein shape of the bound comes from the matrix Laplace-transform method. The point of the result is that $\hat d_{\text{int}}^{+}$ replaces the ensemble size $m$ inside the logarithm.

Because a single ball constrains $S$ in operator norm, every functional that is Lipschitz in $S$ inherits a guarantee that holds simultaneously with all the others. This covers the effective number of voters, the leftover error of model $i$ after accounting for a chosen subset (the Schur complement), and which spectral cluster a model belongs to. A model is declared redundant only if its leftover-error upper bound falls below a tolerance $\epsilon$.

Where This Sits

The redundancy literature supplies the motivation and the targets.

  • Hidden Clones finds the collapse from 17 models to 2–4 effective voters using spectral clustering of an empirical error-correlation matrix, then repairs voting with Hierarchical Family Voting. It treats the collapse purely observationally; our work asks whether that collapse and those clusters are statistically certified at the sample sizes used.
  • Correlated Errors in Large Language Models shows that error agreement rises with accuracy across roughly 350 models. This is the regime where a bound depending on $m$ hurts most and a bound depending on $d_{\text{int}}$ helps most.
  • Diversity is the strength of the AI crowd ranks models by replaceability (Grok 4 at about 1.7 Baseline Points against under 0.1 for GPT-5) on only 113 questions, with no uncertainty attached. It is our small-sample stress case.
  • Ensembling Tabular Foundation Models reports a mean pairwise Q-statistic of 0.961 and recommends greedy selection as a practical default.
  • Quantifying Correlations of Machine Learning Models and What is being transferred in transfer learning? explain why low intrinsic dimension should be the rule. Models in the same algorithmic family, or fine-tuned from the same pretrained weights, share failure modes.

The concentration literature supplies the method.

We do not claim the full general unification of empirical and intrinsic-dimension bounds. Our setting is narrower and more tractable: every summand $x_j x_j^\top$ is positive semidefinite with a known trace bound. We do not use that paper's subset-selection algorithm, because pruning is a principal-submatrix problem, not a sum-of-matrices augmentation.

Research Program

The technical core is turning the oracle intrinsic-dimension bound into an empirical one using two cheap scalar certificates and one matrix certificate, combined by a union bound.

The trace $\mathrm{Tr}(S) = \mathbb{E}\|x\|^2$ is a bounded scalar mean. A scalar empirical Bernstein bound gives an upper confidence value for it.

A valid lower confidence value for $\lambda_{\max}(S)$ comes from sample splitting. We estimate the top eigenvector $v$ on one half of the questions. Since $\lambda_{\max}(S) \ge v^\top S v$ for any unit vector $v$, a scalar bound on the held-out half certifies $v^\top S v$ from below. Dividing the first quantity by the second gives $\hat d_{\text{int}}^{+}$.

The variance proxy $\hat\sigma^2$ uses the plug-in construction of the sharp empirical Bernstein paper, with its asymmetric split of the failure probability, with $\hat d_{\text{int}}^{+}$ substituted for $d$. We will prove that the combined ball has coverage at least $1-\delta$. We will also characterize when it beats the ambient version: roughly when $\log(m/\hat d_{\text{int}}^{+})$ exceeds the price paid for splitting the sample and for the extra union-bound terms.

Two known weaknesses of the oracle bound need attention.

  • Small deviations. The intrinsic-dimension bound has fatter tails than the classical $d\,e^{-t}$ bound at small deviations $t$. We will report the minimum of the two certified widths, which remains valid at the cost of one extra union-bound term.
  • The free-probability gap. That gap concerns matrices with independent entries. It does not apply here, because our randomness lies in independent questions (rows), not independent matrix entries.

On top of the ball we build three certified quantities.

  • Effective size. An interval for $d_{\text{int}}$ and for the participation ratio $\mathrm{Tr}(S)^2/\|S\|_F^2$.
  • Redundancy test. A test that model $i$ is redundant given a set $T$. The ball, together with the λ_min Chernoff lower bound on $S_{TT}$, propagates to an upper bound on the Schur complement $S_{ii} - S_{iT}S_{TT}^{-1}S_{Ti}$.
  • Cluster stability. A check of spectral-cluster membership: a model's assignment is certified when the eigengap exceeds a multiple of the ball radius.

The pruning rule greedily removes certified-redundant models and stops when no removal can be certified. Its guarantee is that, with probability $1-\delta$, no removed model carried more than $\epsilon$ of unexplained error mass.

Baselines

The naive reference point is what the field does now. Point estimates of the error matrix are fed directly into spectral clustering, effective-dimension counts, and replaceability scores, as in Hidden Clones and the AI-crowd paper. It costs nothing and has no coverage guarantee. We measure how often its conclusions flip between independent subsamples.

The strongest competitor for the certification question is the nonparametric bootstrap over questions. It resamples questions, recomputes each functional, and takes percentile intervals. It is the obvious tool a practitioner would reach for, and it adapts to the data automatically. We will run it with the same subsamples, 2,000 resamples, and one interval per functional. A second variant corrects for simultaneity across all functionals with a max-statistic. We beat it if our intervals reach nominal coverage where the bootstrap undercovers, at small $n$ or small eigengaps, while being no more than about 1.5 times wider at large $n$. If the bootstrap covers at nominal rates everywhere and is narrower, our method loses on practicality, and we will say so.

The strongest existing finite-sample method is the sharp matrix empirical Bernstein bound, in both its plug-in and supermartingale forms, applied to $\hat S$ with its $\log m$ factor. On its own synthetic benchmark, the supermartingale version gives a confidence set 10.5% larger than the oracle at $n=1{,}000$ and 2.2% larger at $n=10{,}000$. The plug-in version is 58.9% larger at $n=1{,}000$ (Sharp Matrix Empirical Bernstein Inequalities). No reported result was found on Lacuna for any matrix concentration bound applied to model error matrices. We run it on the identical $x_j$ and $\delta$. We beat it if our width is smaller at the same coverage on pools with $m \gg d_{\text{int}}$.

The oracle intrinsic-dimension bound of Hsu and coauthors is the upper reference. It is computed with the population $S$, taken from the full benchmark, and shows how much the empirical certificates cost.

The key ablation is our exact procedure with $m$ substituted for $\hat d_{\text{int}}^{+}$. It isolates the intrinsic-dimension ingredient from the empirical-variance machinery.

For downstream pruning there are three competitors.

  • Greedy forward selection on validation accuracy. This is the default from the tabular ensembling paper; no standalone reported number was found on Lacuna for greedy selection. The reported reference point there is Cascade Stacking at +0.18% over TabICLv2 for 253 times the compute.
  • Hierarchical Family Voting. It is reported to recover 18–26 percentage points on the Misleading tier (Hidden Clones).
  • Optimal Weight and Inverse Surprising Popularity voting. These are reported to gain 3.36% over majority voting on MMLU (Beyond Majority Voting).

All receive the same validation questions. Our pruning beats them if, at matched final ensemble size, held-out accuracy is equal or better and the pruned set is more stable across subsamples.

Why not just…?

Why not just bootstrap the correlation matrix? This is the real threat. If the bootstrap covers well at the sample sizes people actually use, the theoretical machinery adds little beyond worst-case reassurance. The theoretical motivation for doubting it is that clustering and effective-dimension counts depend on eigengaps, which are non-smooth functionals and exactly where resampling intervals tend to misbehave. But that is a hypothesis, not a cited result. The coverage experiment below settles it directly, because the full benchmarks give us the true $S$. A tie at large $n$ with a bootstrap failure at $n \le 200$ would justify the method. A bootstrap win everywhere would not.

Why not just use greedy selection on validation accuracy? The tabular paper finds greedy selection matches far costlier stacking without hurting calibration. But the forecasting study shows that ranking by solo accuracy misses the least-replaceable model, Grok 4. Greedy selection also gives no guarantee against overfitting 113 questions. We run it head-to-head. If it matches our pruning in held-out accuracy and in stability across subsamples, our contribution is the certificate, not a better pruning rule.

Why not just run spectral clustering and keep one model per family? Hidden Clones shows this works on a pool dominated by one family, but needed hand tuning. Our method is not a replacement for clustering. It is the test of whether a given clustering is supported by the data. The measurement is how often uncertified cluster assignments change across subsamples, compared against the assignments our eigengap test certifies.

Why not just use the standard $\log m$ bound, since $\log 17$ is small? For $m=17$ this objection is largely right. The expected gain at $\delta = 0.05$ is under 1.3 times in width. The case for intrinsic dimension rests on large or unbounded pools: the 349 MMLU models, or pools of checkpoints and prompt variants. In those, $\log(m/\delta)$ roughly doubles relative to $\log(d_{\text{int}}/\delta)$, cutting the needed sample size about in half. The ablation measures this directly. If the gain stays under 10% even at $m = 349$, we will report the empirical certification as the contribution and drop the intrinsic-dimension claim.

Proposed Experiments

The central experiment treats large benchmarks as populations. On MMLU, the full 349 models by 12,032 questions defines the true $S$. On HELM, 71 models by 14,042 questions does the same. We draw 1,000 independent question subsamples at each of $n \in \{50, 100, 200, 500, 1000, 2000\}$ and vary the pool size $m$ from 5 to 349 by sampling models. Diverse pools are drawn across providers; redundant pools are drawn within a single family. The metrics are coverage of $\|\hat S - S\|$ and of each derived functional, interval width, and the fraction of redundancy and cluster decisions that are both certified and correct. Comparisons cover all certification baselines and the ablation. The decisive checks are two. First, our ball must keep nominal coverage at every $(n, m)$. Second, its width advantage over the $\log m$ bound must grow with $m/d_{\text{int}}$ as the bound predicts; if it does not, the core claim is falsified.

The second experiment regenerates answers from open VLMs across several families on VQAv2, TextVQA, and GQA, mirroring the Hidden Clones pool. It asks whether "17 models are worth 2 to 4 voters" is certifiable at the validation sizes used. It also compares certified pruning followed by plain voting against Hierarchical Family Voting, Learned Candidate Scoring, and greedy selection. The comparison metrics are overall accuracy, Misleading-tier accuracy, and the reported GQA reference of 66.47% for Learned Candidate Scoring.

The third experiment reruns the tabular setting on the OpenML-CC18, TALENT, and TabZilla tasks from the tabular ensembling paper. There we test whether certified pruning reproduces the "diversity ceiling" with an explicit guarantee and avoids the calibration loss seen with stacking.

The final check is the Metaculus Q2 2025 set of 113 questions. We ask whether Grok 4's lead in replaceability survives a 95% certificate. Either answer is informative. If it survives, a published conclusion becomes a certified one. If it does not, a widely repeated conclusion turns out to be underdetermined by its data.