An Empirical Investigation of Statistical Significance in NLP

Taylor Berg-Kirkpatrick, David Burkett, Dan KleinEMNLP 2012 · OpenReview

Abstract

We investigate two aspects of the empirical behavior of paired significance tests for NLP systems. First, when one system appears to outperform another, how does significance level relate in practice to the magnitude of the gain, to the size of the test set, and to the similarity of the systems? We explore these issues across a range of NLP tasks using both large collections of past systems' outputs and variants of single systems. Next, we examine how well standard notions of significance hold up under domain shifts, where future distributions are neither independent nor identically distributed.

Introduction

In Natural Language Processing (NLP) research, statistical significance testing—specifically the paired bootstrap—is the standard for demonstrating that a new system outperforms a baseline. Over time, the community has developed informal "rules of thumb," such as the belief that a 0.5 BLEU score increase is typically significant. However, these heuristics are rarely tested empirically. This paper investigates the practical properties of these tests across multiple tasks, asking whether metric gains reliably predict significance and if significance on a test set translates to real-world performance.

Methodology

To map the relationship between metric gain and $p$-values, the study employs two primary comparison methods:

  1. Natural Comparisons: Utilizing outputs from systems submitted to public shared tasks (e.g., WMT, CoNLL) to perform pairwise significance tests.
  2. Synthetic Comparisons: To simulate the incremental changes typical of research, the authors trained dozens of variants of base model architectures (like Moses or the Berkeley Parser) by resampling training data. This allowed for the comparison of highly correlated systems (variants of the same model) against uncorrelated ones (entirely different models).

The Impact of System Correlation

A critical finding is that system correlation significantly influences significance thresholds. When comparing two similar systems—such as a baseline and a version with one additional feature—their outputs are highly correlated. Because they share many identical predictions, a much smaller metric gain is required to achieve statistical significance compared to a comparison between two fundamentally different architectures.

Figure 6: Relationship between F1 gain and significance level (1 - p-value) for similar vs. different model types in constituency parsing.

As shown in Figure 6, for constituency parsing, the "Same model type" curve (green) reaches high confidence levels at lower F1 gains than the "Different model types" curve (red).

The Fragility of Metric "Rules of Thumb"

While it is possible to calculate thresholds where a gain is 95% likely to be significant, these thresholds are highly sensitive to test set size and system similarity. For instance, in the TAC 2008 Summarization task, 95% of borderline significant differences ($p \approx 0.05$) occurred with a ROUGE gain of less than 1.10.

Figure 1: ROUGE gain vs. confidence level for summarization systems. Systems from the same research group (green) often require smaller gains for significance than those from different groups (red).

Figure 1 illustrates this for summarization: systems from the same research group (a proxy for similarity) tend to cluster closer to the y-axis, achieving significance with smaller ROUGE improvements.

Domain Shift: The Ultimate Reality Check

Significance tests are well-calibrated for data from the same distribution, but they fail to predict performance on out-of-domain data. In constituency parsing, a $p$-value near 0.05 on the in-domain WSJ Section 23 correctly predicted the better system on another in-domain set 83% of the time. However, it predicted the better system on the out-of-domain Brown corpus only 48% of the time—worse than a coin flip. To achieve even a 73% reliability on the Brown corpus, the in-domain $p$-value had to be extremely strict ($p < 0.00125$).

Conclusion

Statistical significance should be viewed as a minimum necessary condition for performance claims, not a guarantee of universal superiority. Researchers must remain cautious of informal metric thresholds, as factors like system correlation and domain shift can render them misleading. The study advocates for continued rigorous testing while acknowledging the limitations of i.i.d. assumptions in practical NLP deployment.

Summary updated

Abstract

We investigate two aspects of the empirical behavior of paired significance tests for NLP systems. First, when one system appears to outperform another, how does significance level relate in practice to the magnitude of the gain, to the size of the test set, to the similarity of the systems, and so on? Is it true that for each task there is a gain which roughly implies significance? We explore these issues across a range of NLP tasks using both large collections of past systems' outputs and variants of single systems. Next, once significance levels are computed, how well does the standard i.i.d. notion of significance hold up in practical settings where future distributions are neither independent nor identically distributed, such as across domains? We explore this question using a range of test set variations for constituency parsing.
BibTeX
@inproceedings{bergkirkpatrick2012empirical,
  author = {Taylor Berg-Kirkpatrick and David Burkett and Dan Klein},
  title = {{An Empirical Investigation of Statistical Significance in NLP}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2012},
  url = {https://openreview.net/forum?id=BkbHNzzO-r},
}
Figure