Researchers increasingly use partial correlation to audit how AI systems are evaluated. Partial correlation measures how strongly two quantities move together after a third has been "held constant" by regressing it out. Typical questions are whether an automatic summary metric tracks coherence or only appears to because coherent summaries also tend to be relevant, and whether a model's skill at judging answers is tied to its skill at producing them once answer correctness is accounted for. These audits treat every input as exact. In practice each one is a noisy measurement: a single human rating, a single sampled verdict from a language-model judge, or a correctness label from an automatic answer extractor. Noise in the variable being held constant leaves part of its influence in place. Noise in the variables being correlated pulls the result toward zero. The published conclusions can therefore be wrong in either direction, and nobody currently checks.
We propose a correction that estimates the measurement noise directly from repeated measurements of the same item. These can be several annotators, several sampled judge calls, or several extraction passes. The method subtracts the full noise covariance from the observed covariance before computing partial correlations. Classical correction for attenuation adjusts each variable separately using a scalar reliability. Our correction also estimates correlated errors, such as one annotator's general impression spilling into all the quality criteria they rate at once. Correlated errors of this kind can push a partial correlation away from zero, which is the opposite of attenuation. We validate the correction with a held-out-replicate protocol and then reanalyze two published audits. Both audits use benchmarks in our file and both reach strong conclusions from partial correlations.
Core Claim
When the conditioning variable and the target variables in an evaluation audit are each observed with item-level replicates, the bias of the naive partial correlation can be removed by a moment correction. The correction uses a noise covariance estimated from within-item disagreement. The size of this correction is large enough to change conclusions drawn in existing work. We make two specific predictions. First, in SummEval, the reported collapse of metric-coherence correlation after controlling for relevance understates the true collapse, because relevance is itself a noisy rating. Second, in the self-reference judge study, part of the gap between weak and strong generation-judgment partial correlations reflects attenuation from single-sample judge verdicts.
Write $w_{ik}$ for the vector of all measured variables (for example a metric score, coherence, relevance) on item $i$ from replicate $k$, with $K$ replicates per item and $n$ items. Assume $w_{ik} = z_i + e_{ik}$, where $z_i$ is the true item-level value and the error $e_{ik}$ has mean zero, covariance $\Omega$, and is independent of $z_i$ and across replicates. The within-item scatter then estimates $\Omega$ without knowing $z$:
$$\hat\Omega = \frac{1}{n(K-1)}\sum_{i=1}^{n}\sum_{k=1}^{K}(w_{ik}-\bar w_i)(w_{ik}-\bar w_i)^\top .$$
The covariance of the replicate means, $S$, satisfies $E[S] = \Sigma + \Omega/K$, so the corrected covariance is $\hat\Sigma = S - \hat\Omega/K$. With $P = \hat\Sigma^{-1}$, the corrected partial correlation of variables $x$ and $y$ given the rest is $\rho_{xy\cdot\text{rest}} = -P_{xy}/\sqrt{P_{xx}P_{yy}}$. The off-diagonal entries of $\hat\Omega$ carry the correlated-error component that scalar reliability corrections discard. Item-level bootstrap intervals capture the extra variance of the subtraction. When $\hat\Sigma$ is not positive definite, we project it onto the nearest positive definite matrix and report how often this happens.
The correction removes only noise that varies across replicates. A bias shared by all samples from one judge model is invisible to it. An example is the "bias towards 'correct'" noted in Do Before You Judge. We treat this as a stated limit, not something to fix. Where it matters, we measure it separately by comparing judges.
Relation to Prior Work
The motivating mechanism comes from causal inference. Proximal Causal Learning with Kernels shows that adjusting for a noisy proxy of a confounder fails to block the confounding path. It addresses this with two families of proxies and a kernel-based bridge function. We face the same leak inside linear residualization. Instead of proxies satisfying cross-independence conditions, we use replicates, which evaluation data often contain for free. We target a partial correlation coefficient, not a causal effect.
Three lines of measurement-error work constrain the design and are not directly reusable. Latent-variable estimators of dose-response, VF-VCNet and CEME, model noise only in a single treatment and need large samples or nonlinearity. Evaluation audits have neither: they involve hundreds to a few thousand items, the noise sits in several variables at once, and the quantities of interest are linear. Matrix factorization for noisy covariates recovers a confounder subspace only up to a linear transformation. That is fine for effect estimation but loses the identity of individual criteria, which an audit must keep. The errors-in-variables discovery methods (Yang et al., LV-SEM-ME) rely on non-Gaussian noise and independent component analysis. Bounded, roughly symmetric rating noise offers little of that signal, so we deliberately use only second moments plus replicates.
The applications supply the claims we test. Re-Examining Summarization Evaluation buckets SummEval summaries by human relevance. It reports a Coherence-Relevance correlation of 0.69 and a median relative drop of 0.82 in metric-coherence correlation after bucketing. The relevance it buckets on is itself a noisy rating, and the paper names multiple confounders as an open limitation. Do Before You Judge partials out agent correctness and finds generation-judgment partial correlations "often below 0.3" with chain-of-thought prompting, against 22 of 33 cases above 0.5 with self-reference. It treats each judge verdict and correctness label as exact. Our work does not dispute either method. It asks how much of each reported number survives once measurement noise is modeled.
Research Program
The program has three parts. The first is the estimator and its inference. This covers the replicate-based $\hat\Omega$, a version that handles unequal replicate counts per item, item-level bootstrap intervals, the positive-definite projection, and binary variables. For binary variables we use replicate means of verdicts as continuous proportions and also report a variant built on tetrachoric correlations. We will derive when correlated errors flip the sign of the naive bias. In a three-variable system with conditioning variable $c$, the leak from noise in $c$ inflates the partial correlation in the direction of the confounded association. Error correlation between $x$ and $y$ adds directly to their partial covariance. The two effects can cancel or compound, and the derivation will say which regime each audit sits in.
The second part is a validation protocol that needs no ground truth. We split the replicates. The corrected estimate is computed from a small subset, for example $K=2$, and compared with the naive estimate computed from the mean of many replicates. That mean has noise variance $\Omega/K_{\text{large}}$ and approaches the noise-free value. For language-model judges we can sample many replicates cheaply. For SummEval we use the per-annotator ratings, and we also run semi-synthetic versions in which averaged ratings are treated as truth and noise with controlled reliability and controlled cross-criterion error correlation is injected.
The third part is reanalysis. We recompute the SummEval relevance-controlled metric correlations and the self-reference generation-judgment partial correlations, reporting naive and corrected values with intervals.
Baselines
The naive reference point is the standard two-step residualization on single measurements, exactly as in the audited papers. It reproduces their reported figures: SummEval Coherence-Relevance correlation of 0.69 and a 0.82 median relative drop under bucketing in Re-Examining Summarization Evaluation, and chain-of-thought partial correlations often below 0.3 in Do Before You Judge. It runs on the same items and the same single replicate that our method's subset uses. Beating it means a smaller error against the many-replicate reference, at matched replicate count.
The relevance-bucketing approach of Re-Examining Summarization Evaluation is the existing non-parametric alternative for this question. We use the same bucket boundaries and the same metrics. It is the strongest competitor in the setting the field already uses, because it is assumption-light about functional form. We expect to beat it when relevance is noisy, since buckets defined on a noisy rating mix items of different true relevance.
The strongest principled competitor is classical disattenuation: correcting each correlation by per-variable reliabilities, estimated from the same replicates, before forming the partial correlation. Mathematically this is our estimator with $\hat\Omega$ forced to be diagonal, so it is also the ablation that isolates our key ingredient. It receives identical data and replicates. If the full-covariance version does not beat it on SummEval or in the injected-correlation semi-synthetic runs, our contribution reduces to applying a known correction, and we will report that. No reported result was found on Lacuna for disattenuated partial correlation on SummEval or on the self-reference judge benchmarks.
We also fit a confirmatory latent-factor model, a structural equation model with one latent factor per quality criterion and replicates as indicators, with correlated residuals allowed. It sees the same data and receives the same specification effort.
Why not just…?
Why not just average more annotations or more judge samples? Where this is affordable, it is the right answer, and our validation protocol uses it as the reference. The issue is that published audits and most human-rated benchmarks have few ratings per item, and new human ratings are expensive. The correction is valuable exactly when it reproduces the many-replicate answer from two or three replicates. The held-out-replicate experiment measures this directly. If $K=2$ correction is no closer to the reference than the naive $K=2$ estimate, the method has no reason to exist.
Why not just use classical reliability correction? This is the real threat. It is our method with error correlations set to zero. It wins whenever errors across criteria are uncorrelated. Our case rests on the claim that a rater's general impression bleeds into several criteria rated at the same time, as happens in SummEval. The diagonal-versus-full ablation on real per-annotator data settles this. If the estimated cross-criterion error correlations are near zero, we will say the classical correction suffices and recast the work as an empirical reanalysis.
Why not just bucket or stratify on the confounder? Stratifying on a noisy rating is still conditioning on a proxy, so it inherits the leak described in Proximal Causal Learning with Kernels. It also cannot handle several continuous confounders at once, which the SummEval authors list as unaddressed. The semi-synthetic sweep over relevance reliability will show how bucketing error grows as reliability falls.
Why not just fit a latent-variable structural equation model? In the linear Gaussian case, a correctly specified model estimates the same quantity, so we expect agreement when it is well specified. The moment estimator needs no choice of factor structure, gives partial correlations among the observed criteria directly, and does not fail to converge on small item sets. We will report whether the two agree and how often the latent-factor fit fails or needs re-specification.
Proposed Experiments
The synthetic study draws true item values from multivariate Gaussian and bounded ordinal distributions. It varies sample size (hundreds to a few thousand items), replicate count (two to twenty), reliability, and cross-variable error correlation. It reports bias, root mean squared error, and interval coverage of each estimator against the known partial correlation. The positive-definite projection rate is reported as a diagnostic.
On SummEval we use the per-annotator expert ratings as replicates. We recompute metric-coherence and metric-fluency correlations for ROUGE-1 and ROUGE-2 conditioned on relevance, using naive, bucketed, diagonal-corrected, full-corrected, and latent-factor estimates. We also estimate the cross-criterion error correlation, especially between the coherence and relevance errors, with a bootstrap interval. The decisive check is whether that interval excludes zero and whether the full correction moves the relevance-controlled metric-coherence correlation by more than its own interval width relative to the diagonal correction.
For the judge study we follow the setup of Do Before You Judge on MMLU Pro and GSM8K. We sample each judge verdict and each generated answer repeatedly, compute partial correlations from single samples and from the corrected estimator, and compare both to the many-sample reference. The decisive check is the chain-of-thought condition. If the corrected value stays below 0.3 while the self-reference value stays above 0.5, the original conclusion is robust to noise. If the corrected chain-of-thought value rises substantially toward the self-reference value, part of the reported gain of about 0.35 came from more consistent judging, not from any stronger link between generation and judgment. Either result is informative, and we commit to reporting it.
References
- Do Before You Judge: Self-Reference as a Pathway to Better LLM Evaluation
- Proximal Causal Learning with Kernels: Two-Stage Estimation and Moment Restriction
- A Variational Framework for Estimating Continuous Treatment Effects with Measurement Error
- Identifiable Causal Inference with Noisy Treatment and No Side Information
- Causal Inference with Noisy and Missing Covariates via Matrix Factorization
- Causal Discovery in Linear Latent Variable Models Subject to Measurement Error
- Causal Discovery in Linear Models with Unobserved Variables and Measurement Error
- Re-Examining Summarization Evaluation across Multiple Quality Criteria