In fields like natural language processing and computer vision, we often find ourselves in a "leaderboard" culture. We develop a new metric—say, a better version of ROUGE for summarization or a new embedding-based distance for image captioning—and we want to prove it is superior to the status quo. To do this, we calculate how well our metric correlates with human judgments and compare that to the correlation of an existing baseline.
However, a recurring body of research, led prominently by Yvette Graham and colleagues, argues that simply looking at which correlation coefficient is higher is mathematically insufficient. Because these metrics are being evaluated on the same set of human judgments, the resulting correlations are statistically dependent. Failing to account for this dependency leads to "false ties" or, worse, the crowning of superior-looking metrics that are actually indistinguishable from noise.
The Problem of Overlapping Correlations
Imagine you have two automated metrics, $M_1$ and $M_2$, and a set of human "gold standard" scores, $H$. You calculate the Pearson correlation between $M_1$ and $H$ ($r_{M_1H}$) and between $M_2$ and $H$ ($r_{M_2H}$).
If $r_{M_1H} = 0.85$ and $r_{M_2H} = 0.82$, it is tempting to conclude that $M_1$ is the better metric. But these two values are not independent observations. They are both "looking" at the same human data ($H$). Furthermore, $M_1$ and $M_2$ are often highly correlated with each other ($r_{M_1M_2}$), as they might both rely on similar n-gram matching or embedding logic.
As Qun Liu and Yvette Graham point out in Achieving Accurate Conclusions in Evaluation of Automatic Machine Translation Metrics, standard confidence intervals for a single correlation do not apply here. To determine if $M_1$ is truly better than $M_2$, we need to test the significance of the difference between these two dependent correlations.
The Williams Test as a Rigorous Auditor
The primary tool identified across this research for solving this problem is the Williams test. This statistical test is specifically designed to compare two correlations that share a common variable (in this case, the human ground truth).
The intuition behind the Williams test is that it accounts for the "triangular" relationship between the three variables. If $M_1$ and $M_2$ are very similar to each other, even a small lead in correlation with $H$ might be statistically significant. Conversely, if they are very different, we require a much larger gap to be sure one is better.
This rigorous auditing has overturned several "common sense" beliefs in the field:
- Summarization: In Re-evaluating Automatic Summarization with BLEU and 192 Shades of ROUGE, Yvette Graham showed that the machine translation metric BLEU actually performs as well as or better than ROUGE when subjected to the Williams test, challenging years of standard practice.
- Image Captioning: Aykut Erdem and Mert Kilickaya used the test in Re-evaluating Automatic Metrics for Image Captioning to prove that Word Mover’s Distance (WMD) provides a statistically significant improvement over traditional exact-match metrics.
- Grammatical Error Correction: Shamil Chollampatt and Hwee Tou Ng argued in A Reassessment of Reference-Based Grammatical Error Correction Metrics that the field's shift to new metrics was often premature, as those metrics failed to significantly outperform baselines under proper statistical scrutiny.
Beyond NLP: Alignment and Biomarkers
The need for this statistical framework extends beyond evaluating translation or summarization. It is a fundamental requirement whenever we compare how well two different models or signals align with a single reference.
In the study of Large Language Models, Shanshan Xu and Yanai Elazar applied the Williams test to determine if a model's political leanings were significantly more aligned with its training data than with human survey respondents. Without the test, a researcher might see two high correlation coefficients and incorrectly assume the model is "aligned" with humans, when in fact the correlation with training data is statistically dominant.
Even in clinical research, as seen in Jing Feng et al.'s work on preoperative plasma D-Dimer levels, the test is used to compare a novel biomarker's predictive power against established standards.
Requirements for Robust Evaluation
For the Williams test to be effective, the underlying data must be high-quality. Timothy Baldwin, Yvette Graham, and Nitika Mathur demonstrated in Accurate Evaluation of Segment-level Machine Translation Metrics that moving from noisy, relative human rankings to standardized continuous scales is essential. Continuous scores allow for the calculation of Pearson correlations, which in turn enables the Williams test to provide a clear, statistically grounded leaderboard.
As models become more similar and performance gains become more marginal, the work of Aaditya Ramdas and Ilmun Kim on confidence sets for the best model suggests that accounting for these dependencies is no longer optional—it is the only way to ensure that the "best" model is actually the best.