Abstract
We investigate two aspects of the empirical behavior of paired significance tests for NLP systems. First, when one system appears to outperform another, how does significance level relate in practice to the magnitude of the gain, to the size of the test set, and to the similarity of the systems? We explore these issues across a range of NLP tasks using both large collections of past systems' outputs and variants of single systems. Next, we examine how well standard notions of significance hold up under domain shifts, where future distributions are neither independent nor identically distributed.
Introduction
In Natural Language Processing (NLP) research, statistical significance testing—specifically the paired bootstrap—is the standard for demonstrating that a new system outperforms a baseline. Over time, the community has developed informal "rules of thumb," such as the belief that a 0.5 BLEU score increase is typically significant. However, these heuristics are rarely tested empirically. This paper investigates the practical properties of these tests across multiple tasks, asking whether metric gains reliably predict significance and if significance on a test set translates to real-world performance.
Methodology
To map the relationship between metric gain and $p$-values, the study employs two primary comparison methods:
- Natural Comparisons: Utilizing outputs from systems submitted to public shared tasks (e.g., WMT, CoNLL) to perform pairwise significance tests.
- Synthetic Comparisons: To simulate the incremental changes typical of research, the authors trained dozens of variants of base model architectures (like Moses or the Berkeley Parser) by resampling training data. This allowed for the comparison of highly correlated systems (variants of the same model) against uncorrelated ones (entirely different models).
The Impact of System Correlation
A critical finding is that system correlation significantly influences significance thresholds. When comparing two similar systems—such as a baseline and a version with one additional feature—their outputs are highly correlated. Because they share many identical predictions, a much smaller metric gain is required to achieve statistical significance compared to a comparison between two fundamentally different architectures.
As shown in Figure 6, for constituency parsing, the "Same model type" curve (green) reaches high confidence levels at lower F1 gains than the "Different model types" curve (red).
The Fragility of Metric "Rules of Thumb"
While it is possible to calculate thresholds where a gain is 95% likely to be significant, these thresholds are highly sensitive to test set size and system similarity. For instance, in the TAC 2008 Summarization task, 95% of borderline significant differences ($p \approx 0.05$) occurred with a ROUGE gain of less than 1.10.
Figure 1 illustrates this for summarization: systems from the same research group (a proxy for similarity) tend to cluster closer to the y-axis, achieving significance with smaller ROUGE improvements.
Domain Shift: The Ultimate Reality Check
Significance tests are well-calibrated for data from the same distribution, but they fail to predict performance on out-of-domain data. In constituency parsing, a $p$-value near 0.05 on the in-domain WSJ Section 23 correctly predicted the better system on another in-domain set 83% of the time. However, it predicted the better system on the out-of-domain Brown corpus only 48% of the time—worse than a coin flip. To achieve even a 73% reliability on the Brown corpus, the in-domain $p$-value had to be extremely strict ($p < 0.00125$).
Conclusion
Statistical significance should be viewed as a minimum necessary condition for performance claims, not a guarantee of universal superiority. Researchers must remain cautious of informal metric thresholds, as factors like system correlation and domain shift can render them misleading. The study advocates for continued rigorous testing while acknowledging the limitations of i.i.d. assumptions in practical NLP deployment.
Summary updated