Statistical Significance Testing for Dependent Correlations

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 215 papers30 conceptsfrozen 2026-10-04

In fields like natural language processing and computer vision, we often find ourselves in a "leaderboard" culture. We develop a new metric—say, a better version of ROUGE for summarization or a new embedding-based distance for image captioning—and we want to prove it is superior to the status quo. To do this, we calculate how well our metric correlates with human judgments and compare that to the correlation of an existing baseline.

However, a recurring body of research, led prominently by Yvette Graham and colleagues, argues that simply looking at which correlation coefficient is higher is mathematically insufficient. Because these metrics are being evaluated on the same set of human judgments, the resulting correlations are statistically dependent. Failing to account for this dependency leads to "false ties" or, worse, the crowning of superior-looking metrics that are actually indistinguishable from noise.

The Problem of Overlapping Correlations

Imagine you have two automated metrics, $M_1$ and $M_2$, and a set of human "gold standard" scores, $H$. You calculate the Pearson correlation between $M_1$ and $H$ ($r_{M_1H}$) and between $M_2$ and $H$ ($r_{M_2H}$).

If $r_{M_1H} = 0.85$ and $r_{M_2H} = 0.82$, it is tempting to conclude that $M_1$ is the better metric. But these two values are not independent observations. They are both "looking" at the same human data ($H$). Furthermore, $M_1$ and $M_2$ are often highly correlated with each other ($r_{M_1M_2}$), as they might both rely on similar n-gram matching or embedding logic.

As Qun Liu and Yvette Graham point out in Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics, standard confidence intervals for a single correlation do not apply here. To determine if $M_1$ is truly better than $M_2$, we need to test the significance of the difference between these two dependent correlations.

The Williams Test as a Rigorous Auditor

The primary tool identified across this research for solving this problem is the Williams test. This statistical test is specifically designed to compare two correlations that share a common variable (in this case, the human ground truth).

The intuition behind the Williams test is that it accounts for the "triangular" relationship between the three variables. If $M_1$ and $M_2$ are very similar to each other, even a small lead in correlation with $H$ might be statistically significant. Conversely, if they are very different, we require a much larger gap to be sure one is better.

This rigorous auditing has overturned several "common sense" beliefs in the field:

Beyond NLP: Alignment and Biomarkers

The need for this statistical framework extends beyond evaluating translation or summarization. It is a fundamental requirement whenever we compare how well two different models or signals align with a single reference.

In the study of Large Language Models, Shanshan Xu and Yanai Elazar applied the Williams test to determine if a model's political leanings were significantly more aligned with its training data than with human survey respondents. Without the test, a researcher might see two high correlation coefficients and incorrectly assume the model is "aligned" with humans, when in fact the correlation with training data is statistically dominant.

Even in clinical research, as seen in Jing Feng et al.'s work on preoperative plasma D-Dimer levels, the test is used to compare a novel biomarker's predictive power against established standards.

Requirements for Robust Evaluation

For the Williams test to be effective, the underlying data must be high-quality. Timothy Baldwin, Yvette Graham, and Nitika Mathur demonstrated in Accurate Evaluation of Segment-level Machine Translation Metrics that moving from noisy, relative human rankings to standardized continuous scales is essential. Continuous scores allow for the calculation of Pearson correlations, which in turn enables the Williams test to provide a clear, statistically grounded leaderboard.

As models become more similar and performance gains become more marginal, the work of Aaditya Ramdas and Ilmun Kim on confidence sets for the best model suggests that accounting for these dependencies is no longer optional—it is the only way to ensure that the "best" model is actually the best.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 9 papers
  2. Version 1 · 2026-10-04 · 15 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_dependent_correlation_significance_testi_1e9d2176,
  title = {dependent correlation significance testing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dependent-correlation-significance-testing/rdr_1e9d2176fa3049fda316a48065f0806c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_statistical_significance_testing_for_dep_1e9d2176_v1,
  title = {Statistical Significance Testing for Dependent Correlations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/statistical-significance-testing-for-dependent-correlations/rdr_1e9d2176fa3049fda316a48065f0806c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}