Principles from Clinical Research for NLP Model Generalization

Aparna Elangovan, Jiayuan He, Yuan Li, Karin Verspoor

Principles from Clinical Research for NLP Model Generalization

Generalization failure in NLP is frequently attributed to "out-of-distribution" (OOD) data. However, this paper argues that the root cause is often a lack of internal validity—the model never truly learned the underlying task, but instead exploited "shortcuts" or spurious correlations within the training data. By borrowing principles from clinical research, the authors propose a framework to distinguish between true task mastery and surface-level pattern matching.

The Hierarchy of Validity

In clinical trials, internal validity ensures that an observed effect is caused by the treatment rather than confounding factors. Applying this to NLP, the authors argue that internal validity—establishing intra-test set model literacy—is a prerequisite for any form of generalization.

Figure 1: The proposed hierarchy of validity, placing internal validity at the core of all generalization efforts.

As shown in Figure 1, internal validity serves as the foundation. Without ensuring the model has learned the correct linguistic capabilities to produce its performance, claims of cross-domain or cross-task generalization are premature. The community often ignores this "internal" check, jumping straight to external validity (generalization).

Methodology: Detecting Shortcuts via Surrogate Modeling

To investigate model behavior, the authors use Interpretable Surrogate Modeling. They train simple, transparent models (e.g., Multinomial Naive Bayes) using only "surface patterns" like entity distance or sentence length. They then measure the Cohen’s Kappa ($\kappa$) correlation between these surrogates and complex models like BERT. A high correlation indicates the complex model is likely relying on the same simple shortcuts rather than linguistic logic.

Figure 3: Cohen’s Kappa ($\kappa$) correlations between surrogate models (NB, NB-T) and model predictions across different datasets.

Key Results and Shortcut Reliance

The empirical results reveal significant shortcut reliance across multiple benchmarks. As detailed in Figure 3, in the PTM-PPI task, the surrogate model (NB-T) achieved a correlation of $\kappa = 0.73$ with the complex model’s actual predictions on the test set (TS (MP)), which is significantly higher than its correlation with the ground truth ($\kappa = 0.55$). On the generalization set (GH), this correlation rose to a staggering $\kappa = 0.85$. This provides a "smoking gun" that the model is more "in sync" with surface shortcuts (like the distance between entities) than with the actual biological truth.

Similarly, in the SNLI benchmark, surrogate models reached a $\kappa$ of 0.34 by merely observing sentence length and negations, confirming that these heuristics remain a major factor in NLI performance.

Causal Matching and Contrast Sets

To move beyond correlation, the authors suggest adapting "matching" from randomized controlled trials using contrast sets. These are pairs of examples where a minor, label-flipping change is made (e.g., changing "good" to "bad"). If a model's prediction remains unchanged despite a linguistically significant perturbation, it lacks the internal validity required for robust performance. Expert verification on the generalization set showed that only 20% of the model's "correct" predictions were actually right for the right reasons.

Conclusion

The NLP community must move beyond the "OOD excuse." High benchmark scores do not equate to task literacy. The authors call for the standard adoption of internal validity checks, such as surrogate correlation analysis and contrast set evaluation, to ensure models are truly learning linguistic capabilities rather than memorizing dataset-specific signatures.

Summary updated

Abstract

The NLP community typically relies on performance of a model on a held-out test set to assess generalization. Performance drops observed in datasets outside of official test sets are generally attributed to "out-of-distribution'' effects. Here, we explore the foundations of generalizability and study the various factors that affect it, articulating generalizability lessons from clinical studies. In clinical research generalizability depends on (a) internal validity of experiments to ensure controlled measurement of cause and effect, and (b) external validity or transportability of the results to the wider population. We present the need to ensure internal validity when building machine learning models in natural language processing, especially where results may be impacted by spurious correlations in the data. We demonstrate how spurious factors, such as the distance between entities in relation extraction tasks, can affect model internal validity and in turn adversely impact generalization. We also offer guidance on how to analyze generalization failures.
BibTeX
@inproceedings{elangovan2024principles,
  author = {Aparna Elangovan and Jiayuan He and Yuan Li and Karin Verspoor},
  title = {{Principles from Clinical Research for NLP Model Generalization}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.127},
}
Figure