When developing automatic evaluation metrics for tasks like machine translation, text summarization, or grammatical error correction, the standard validation step is to compute the correlation (such as Pearson's $r$) between the metric's scores and human judgments. If Metric A achieves a higher raw correlation than Metric B on a benchmark dataset, researchers frequently conclude that Metric A is superior.
However, Graham et al. demonstrated that this common practice contains a major methodological flaw. A higher raw correlation coefficient does not automatically mean the improvement is statistically significant. Without direct significance testing of the difference between these correlations, marginal gains are often just random noise, leading the field to adopt new metrics that do not actually outperform established baselines.
Why Independence Assumptions Fail
The core statistical challenge in metric comparison stems from data dependency. If you were comparing the performance of Metric A on Dataset 1 and Metric B on Dataset 2, the two resulting correlation coefficients would be independent. But in standard NLP evaluation, we compute the correlation of Metric A against human scores on the same dataset used to evaluate Metric B.
Because both metrics are evaluating the exact same underlying sentences or documents, their outputs are highly correlated with each other. As Liu et al. emphasize, the resulting correlations with human judgment are statistically dependent. Failing to account for this dependency when calculating confidence intervals for the difference between correlations leads to severe errors, frequently resulting in false positives where noise is mistaken for genuine algorithmic improvement.
To build intuition, imagine a triangle of relationships between three variables:
- Metric A
- Metric B
- Human Gold Standard (Y)
We want to know if the correlation between Metric A and Human Judgments ($r_{AY}$) is significantly larger than the correlation between Metric B and Human Judgments ($r_{BY}$). To answer this accurately, we cannot look at $r_{AY}$ and $r_{BY}$ in isolation. We must factor in the third side of the triangle: the correlation between the two metrics themselves ($r_{AB}$).
If Metric A and Metric B are highly correlated ($r_{AB}$ is close to 1), they share a massive amount of variance. This shared variance means that the variance of their difference ($r_{AY} - r_{BY}$) is much smaller than it would be if they were independent. Consequently, even a tiny mathematical difference between $r_{AY}$ and $r_{BY}$ might be highly statistically significant. Conversely, if the metrics are not tightly coupled, a larger raw difference might still be due to chance.
The Williams Test: Factoring in Shared Variance
To resolve this dependency problem, researchers such as Graham and Ng et al. introduced the Williams test to the evaluation of NLP systems. The Williams test is a specialized statistical hypothesis test designed specifically for comparing dependent, overlapping correlation coefficients.
Instead of treating the two correlation calculations as separate events, the Williams test incorporates the full covariance structure among Metric A, Metric B, and the human scores. By explicitly including $r_{AB}$ in the denominator of its $t$-statistic, the test adjusts the threshold of significance based on how much information the two metrics share. This prevents researchers from overestimating the significance of performance gains and provides a mathematically rigorous way to confirm whether one automated metric truly mirrors human judgment better than another.
Broad Applications Across Language and Generative Models
Applying dependent correlation testing has fundamentally shifted how researchers validate evaluation metrics across several domains:
- Machine Translation & Quality Estimation: Graham showed that utilizing significance tests for dependent correlations prevents predictive models from "gaming" standard error metrics, leading to far more reliable system rankings. Similarly, Baldwin et al. demonstrated that applying the Williams test to continuous human scores allows researchers to confidently verify the superiority of segment-level translation metrics across diverse language pairs.
- Text Summarization: By auditing the field with the Williams test, Graham revealed that long-standing assumptions about the ROUGE metric were flawed, proving that proper statistical scrutiny can completely reshape established leaderboards.
- Generative Model Selection: The underlying intuition of accounting for model dependency extends beyond NLP metrics to the evaluation of generative models. When comparing how close two competing generative models are to a single target dataset, the models' outputs are evaluated against the same reference. Belilovsky et al. developed a non-parametric test of relative similarity using Maximum Mean Discrepancy (MMD). By incorporating the covariance between the two models relative to the reference dataset, their approach significantly increases the sensitivity of hypothesis testing, allowing for precise model selection where independent tests would fail.
Key Foundational Work
Testing for Significance of Increased Correlation with Human Judgment (2014) — contributed 8 concept elements to this direction.
A Decade of Automatic Content Evaluation of News Summaries: Reassessing the State of the Art (2013) — contributed 1 concept elements to this direction.