Guaranteed-Conservative Combination of LHC Measurements with Unknown Cross-Correlations

When physicists test a proposed extension of the Standard Model against hundreds of published LHC measurements, they need the correlations between those measurements, and those correlations are usually not published. Tools such as Contur handle this by keeping only the single most sensitive bin or measurement. That choice never falsely excludes a model, but it throws away most of the data. The other option, treating measurements as independent and summing them, uses everything but can exclude models that are actually allowed. We propose a middle path with a mathematical guarantee. Split the covariance into three parts. The first is the within-measurement blocks that experiments do publish. The second is a small number of named systematic effects shared across measurements (luminosity, jet energy scale), which can be written as a low-rank term. The third is an unknown remainder whose size is bounded using event-overlap information computed from simulation. Under this split, the true goodness-of-fit statistic has a provable lower bound, and computing that bound only requires inverting each small block separately.

The numerical side borrows the idea of selective inversion from communication-avoiding linear algebra Communication-Avoiding Parallel Algorithms for Solving Triangular Systems of Linear Equations. That work explicitly inverts small, well-conditioned diagonal blocks so that sequential triangular solves become matrix multiplications. Here the same move makes one evaluation cost roughly one batched matrix multiply per measurement. Memory then grows with the sum of block sizes squared rather than with the square of the total bin count. As a result, millions of (parameter point, pseudo-experiment) evaluations can stream through a single accelerator without ever forming the dense matrix. The main contribution is statistical: exclusions that are guaranteed to be conservative and still use correlated shape information. The memory and inversion structure is what keeps this cheap enough to run across a full parameter scan.

Core Claim

Write the full bin-level covariance of a measurement bank as

$$C = B + UU^\top + E,$$

where $B$ is block-diagonal and its blocks $C_{ii}$ are the published within-measurement covariances. $U$ is an $n \times r$ matrix with one column per named shared systematic ($n$ total bins, $r$ small). $E$ is the unknown cross-measurement remainder, with zero diagonal blocks. After whitening by $B$, let $\rho_{ij}$ bound the spectral norm of block $(i,j)$ of $E$; this is the largest canonical correlation between the leftover noise of measurements $i$ and $j$. Let $\lambda = \max_i \sum_j \rho_{ij}$. A block version of Gershgorin's theorem gives $C \preceq (1+\lambda)(B + UU^\top)$, and therefore, for any signal shift $s$,

$$s^\top C^{-1} s \;\ge\; \max\!\Big(\tfrac{1}{1+\lambda}\, s^\top (B+UU^\top)^{-1} s,\;\; \max_i s_i^\top C_{ii}^{-1} s_i\Big).$$

The second term always holds, because any positive semi-definite matrix with those diagonal blocks satisfies it. It is exactly the "most sensitive measurement" rule, so the bound never does worse than current practice. The first term recovers nearly the full independent-sum sensitivity when $\lambda$ is small. That happens when most cross-talk is captured by the named systematics and measurements overlap sparsely in the events they select.

The inverse $(B+UU^\top)^{-1}$ never needs to be formed. It follows from the precomputed block inverses $C_{ii}^{-1}$ plus one $r \times r$ correction via the Woodbury identity.

The guarantee covers the expected-exclusion statistic, which is quadratic in the signal. The observed $\Delta\chi^2$ also contains a cross term, $-2s^\top C^{-1}(d-b)$, whose sign the bound does not control; here $d$ is the data and $b$ the Standard Model prediction. We treat that cross term empirically below and do not claim a guarantee for it.

Research Program

The first component estimates the overlap bounds $\rho_{ij}$. Contur already passes Standard Model Monte Carlo events through every Rivet analysis. We record, per event, which bins of which measurements it enters. For counting-dominated statistics, bin-pair correlations can be estimated from shared events. A bootstrap over events then gives an upper confidence value for each $\rho_{ij}$, and most pairs from different datasets or disjoint selections come out exactly zero. Systematics that HepData breaks down by named source become columns of $U$. Only systematics that are unnamed or incompatibly defined are pushed into $E$, and each of those is bounded by assuming full correlation of its reported magnitude. This step is the real crux, and its failure mode is clear: if $\lambda$ comes out large, the bound collapses to the most-sensitive-measurement rule, and we will have learned that combination gains depend on better publication practice, as urged by the LHC Reinterpretation Forum.

The second component is the evaluation engine. We compute a Cholesky factor of each $C_{ii}$ once, form $L_{ii}^{-1}$ explicitly, and precompute $W = B^{-1}U$ together with the $r \times r$ capacitance matrix. For a batch of residual columns $R$ (parameter points times pseudo-experiments), the statistic is the sum of column norms of $L_{ii}^{-1}R_i$ minus a rank-$r$ correction. Batches stream from disk or host memory in chunks.

This is the multiple-right-hand-side triangular problem studied by Wicky et al. They note that the benefit of inversion over plain substitution disappears when the number of right-hand sides per processor dominates the matrix size (preprint). Our regime is exactly that one. So we do not claim a communication saving. We claim, and will test, a matrix-multiply throughput and memory advantage over forming and factoring the dense matrix, and a stability advantage from inverting only small, well-conditioned blocks. The engine is built to be the per-point evaluator that sits under an active-learning sampler.

Baselines

The naive reference is Contur's current default: the most sensitive single bin or measurement per analysis pool Contur toolkit. It is conservative by construction and is exactly the second term of our bound, so any gain we report must be excluded area beyond this rule on the same Rivet outputs.

The opposite reference is the independent sum, which sets cross-measurement correlations to zero and uses all bins. Its excluded area is an upper envelope we should approach but not exceed in guarantee.

The strongest competitor on accuracy is the dense truth: the full $n \times n$ covariance built explicitly and factored with a standard dense Cholesky routine. This is possible only where the truth is known, meaning synthetic banks generated with fully specified correlations, plus subsets of measurements whose HepData records include cross-distribution covariances where they exist. It is run on the same residuals, in double precision, with no tuning budget, since it has no hyperparameters. Beating it is impossible by definition. Success means excluded area within a stated fraction of the dense truth, with zero false exclusions relative to it.

On sampling cost, the strongest existing method is the Contur ORACLE. It reports 90% precision and 92% recall using just over 25,000 points (6.3% of the grid) on a scan otherwise needing about 500,000 core-hours. We run the ORACLE with its published settings twice, once with the default statistic and once with ours as the black-box evaluator. The comparison uses the same grid, stopping rule and random seeds, and we report precision and recall against a full-grid dense-truth scan.

Two ablations isolate what matters. Dropping $U$, so that all shared systematics go into $E$, measures how much of the gain comes from naming systematics. Dropping the overlap estimates, so that $\rho_{ij}=1$ for every pair within a dataset, measures how much comes from simulation-based overlap bounds.

For the computational comparison, dense Cholesky with batched triangular solves on the same device, same batch sizes and same precision is the baseline. We report throughput and peak memory as $n$ grows. No reported result was found on Lacuna for dense covariance inversion throughput in any BSM fitting tool.

Why not just…?

Why not just build the full covariance and factor it with LAPACK? For a few thousand bins with fully known correlations, this is the right answer, and our engine gives no accuracy advantage. It appears as the oracle baseline. The obstacle is not the factorization but the missing entries. As the Contur authors document, the correlations are frequently absent, so the "full" matrix cannot be written down without inventing numbers. The computational side matters only at the margin: when $n$ grows, when covariances depend on the parameter point through signal simulation statistics, and when thousands of pseudo-experiments per point are needed, as in the cMSSM exclusion (Killing the cMSSM softly). The throughput experiment will show where, if anywhere, dense factoring stops being adequate. We accept the outcome if the answer is "never at current bank sizes."

Why not just keep the single-bin rule, or assume independence? The single-bin rule discards shape information that From models to SMEFT and back? shows is needed to lift degeneracies between parameters. Independence has no guarantee against false exclusion. Careless combination can also distort conclusions in the other direction: the cMSSM fit's $p$-value moves from $(4.9 \pm 0.7)\%$ to about 50% depending on which observables enter Killing the cMSSM softly. Our bound contains the single-bin rule as a special case, and the decisive experiment measures how much area lies between it and the dense truth.

Why not just fill the unknown blocks with a regularized guess? This is a real threat for point estimates, since a sensible guess may land close to the truth. But a guess carries no guarantee, and reinterpretation results are used to rule models out. We will include a guessed-correlation variant (overlap-estimated correlations plugged in directly, with no bound) and report both its false-exclusion rate and its area against the dense truth. If its false-exclusion rate is negligible across the synthetic stress tests, the bound's value reduces to certification, and we will say so.

Why not just use active learning to evaluate fewer points? The ORACLE reduces how many points are evaluated. It says nothing about whether each evaluation is statistically sound, and it treats the evaluator as a black box, so the two approaches compose. The experiment above tests whether a statistic with more discriminating power changes where the sampler spends its budget, and whether the ORACLE's reported precision and recall survive against a stronger truth.

Proposed Experiments

The decisive check is synthetic. We generate measurement banks whose structure mirrors the Contur analysis pools (block sizes, dataset groupings, named systematics). We plant cross-correlations of controlled strength, including adversarial ones placed along signal directions. We then inject the three ORACLE benchmark models (the simplified dark-matter model with a vector mediator, the 2HDM+a model, and the vector-like quark model). On these banks we measure two things. The first is the false-exclusion rate relative to the dense truth, which must be zero for the expected statistic and is reported for the observed statistic. The second is the excluded parameter area recovered, as a fraction of the gap between the single-bin rule and the dense truth, plotted against $\lambda$.

We then run on the real Contur measurement bank for the same models. There the truth is unknown, so we report excluded area against the single-bin rule, the fitted distribution of $\lambda$ across pools, and agreement with the dense computation on the subsets where cross-distribution covariances are published. Finally, we pair the evaluator with the ORACLE as described in Baselines and profile throughput and peak memory against dense Cholesky as the bin count and batch width grow.

The program succeeds if real pools have small enough $\lambda$ that the guaranteed bound recovers a substantial share of the independent-sum gain. It fails informatively if $\lambda$ is dominated by unnamed systematics, which would give a quantitative case for publishing systematic breakdowns by name.