The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions

Xiang Zhou, Yixin Nie, Hao Tan, Mohit Bansal

Abstract

We find that the performance of state-of-the-art models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable. This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from? (3) How should we handle this instability and what are some potential solutions? For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve. We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine. Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets. Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models.

Introduction

In recent years, the NLP community has moved beyond simple accuracy on benchmarks like SNLI or SQuAD. To see if models truly understand language or are just exploiting statistical shortcuts, researchers use "analysis datasets" (or stress tests) like HANS. These datasets target specific linguistic phenomena, such as whether a model is fooled by simple word overlap.

However, a troubling trend has emerged: a model might perform brilliantly on HANS in one training run and fail miserably in the next, simply by changing the random seed. This "performance instability" suggests that a model’s "understanding" of concepts like negation or numerical reasoning may be a matter of chance. This paper investigates the source of this instability and its implications for model evaluation.

Methods and Trajectory Analysis

The authors conducted an empirical study using four major architectures: BERT, RoBERTa, XLNet, and ESIM. Each model was trained 10 times with different random seeds and evaluated across a suite of NLI and RC datasets.

A key methodological contribution is Trajectory Tracking. Instead of just looking at the final model, the researchers evaluated performance at every checkpoint during training. They found that instability is not merely a "final state" issue but exists throughout the entire training process.

Accuracy trajectories across training iterations for various evaluation sets.

Additionally, the authors performed Variance Decomposition, mathematically splitting the total variance of a model's accuracy into independent variance (randomness in individual predictions) and inter-data covariance (randomness caused by groups of correlated examples).

Key Results: The Magnitude of Instability

The empirical results highlight a significant gap between standard benchmarks and analysis sets.

  • Extreme Variance: On the Numerical subset of the Stress Test (STR-NU), the variance in performance was 27 times higher than on standard development sets.
  • The HANS Gap: For the HANS dataset, the gap between the highest and lowest performing runs of the same model reached 4 percentage points. For the Numerical Stress Test, it was a staggering 10 points.
  • Selection Failure: There is a very low correlation between performance on standard validation sets (like MNLI) and analysis sets. A model that appears superior on a standard benchmark may perform poorly on a stress test.
Accuracy spread across multiple runs for different models and datasets, highlighting the high variance in analysis sets like STR-NU.

The Source: The "Block Effect"

The central insight of the paper is that instability is driven by inter-example correlation. Many analysis datasets are constructed using templates or synthetic rules, leading to large "blocks" of similar examples. If a model’s random initialization causes it to struggle with one specific template, it fails on the entire block, resulting in a massive drop in total accuracy. This "block effect" means that failures are correlated rather than independent, which amplifies variance.

Discussion and Recommendations

The authors conclude that current model-selection routines are often unreliable for analysis sets. They offer several suggestions for the community:

  1. Report Variance: Researchers should stop reporting single-run results. Instead, they should report the mean and variance over multiple runs.
  2. Decomposed Variance: Reporting the decomposed variance (independent vs. covariance) provides more interpretable results and allows for fairer comparisons.
  3. Increase Diversity: Future stress tests should move away from purely template-based construction toward datasets with higher linguistic diversity to reduce the "block effect."
  4. Model Biases: There is a need for models with better inductive biases that can capture structural and logical properties more stably, rather than relying on the "lottery" of random initialization.

Code is available at: https://github.com/owenzx/InstabilityAnalysis

Summary updated

Abstract

We find that the performance of state-of-the-art models on Natural Language Inference (NLI) and Reading Comprehension (RC) analysis/stress sets can be highly unstable. This raises three questions: (1) How will the instability affect the reliability of the conclusions drawn based on these analysis sets? (2) Where does this instability come from? (3) How should we handle this instability and what are some potential solutions? For the first question, we conduct a thorough empirical study over analysis sets and find that in addition to the unstable final performance, the instability exists all along the training curve. We also observe lower-than-expected correlations between the analysis validation set and standard validation set, questioning the effectiveness of the current model-selection routine. Next, to answer the second question, we give both theoretical explanations and empirical evidence regarding the source of the instability, demonstrating that the instability mainly comes from high inter-example correlations within analysis sets. Finally, for the third question, we discuss an initial attempt to mitigate the instability and suggest guidelines for future work such as reporting the decomposed variance for more interpretable results and fair comparison across models. Our code is publicly available at: https://github.com/owenzx/InstabilityAnalysis
BibTeX
@inproceedings{zhou2020curse,
  author = {Xiang Zhou and Yixin Nie and Hao Tan and Mohit Bansal},
  title = {{The Curse of Performance Instability in Analysis Datasets: Consequences, Source, and Suggestions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.659},
}
Figure