Disentangling Spurious Correlations: How IRM Handles Diverse Shortcuts in Text Classification

Paul Grunenwald, Benoît Crabbé, Guillaume Wisniewski, Pirmin LembergerarXiv 2026 · DOI · arXiv

TL;DR

This paper provides a systematic evaluation of Invariant Risk Minimization (IRM) in NLP, demonstrating its ability to reject spurious shortcuts across diverse causal mechanisms. By leveraging multiple training environments, IRM identifies robust linguistic features that generalize to out-of-distribution (OOD) data, significantly outperforming standard training methods when dataset-specific biases are inverted.

What Problem Does This Paper Address?

Modern NLP models are notoriously "lazy." Under the standard Empirical Risk Minimization (ERM) paradigm, models often latch onto the easiest statistical signal to minimize training loss. These "shortcuts"—such as negation words in sentiment analysis or dialectal markers—might work well on a specific dataset but fail catastrophically when the model encounters a real-world distribution shift where those correlations no longer hold.

While Invariant Risk Minimization (IRM) has been proposed as a theoretical solution to find features that remain stable across different data "environments," its practical utility in the messy world of natural language has been a subject of debate. This paper seeks to clarify when and how IRM works for text classification by testing it against various types of spurious correlations.

Methods

The authors evaluate IRMv1, a practical relaxation of the original IRM objective. The core idea is to find a feature representation $\Phi$ such that the optimal classifier built on top of it is the same across all training environments.

To rigorously test this, they use a causality-driven framework to inject three distinct types of shortcuts into text datasets:

  1. Label-Generated Features ($Y \to Z \to X$): The label itself generates a latent concept that manifests as a textual artifact (e.g., a specific token).
  2. Confounding ($Y \leftarrow U \to Z \to X$): An unobserved variable influences both the label and a textual feature, creating a "backdoor path" correlation.
  3. Selection Bias ($Y \to S \leftarrow Z \to X$): Data is subsampled based on a "collider" variable $S$ that depends on both the label and a feature, inducing a non-causal correlation in the observed data.
Optimization dynamics: ERM learns a global shortcut, while IRM uses environment boundaries to reject it.

The experiments use a frozen DistilBERT encoder with a trainable linear head, ensuring that any performance difference is due to the optimization objective (ERM vs. IRM) rather than model capacity or fine-tuning dynamics.

Datasets

The study spans both synthetic and realistic settings:

  • Synthetic Benchmarks: Amazon Books (Causal), AG News (Causal), and IMDB Genres (Anti-causal).
  • Bias in Bios: A dataset of professional biographies used to study naturally occurring gender-profession associations.
  • ActorCorr: A sentiment analysis task where specific actor names (e.g., Morgan Freeman vs. Adam Sandler) are injected as semantically "loaded" shortcuts.

Key Insights and Ideas

  • Structural Agnosticism: IRM successfully rejects shortcuts regardless of whether they arise from confounding, selection bias, or label-generation. This suggests its effectiveness depends more on environment diversity than the specific causal origin of the bias.
  • Sensitivity to Disparity: IRM is remarkably sensitive. It can leverage even a slight statistical disparity between environments to begin discarding a shortcut in favor of a stable, task-relevant signal.
  • Robustness to Noise: As the genuine task signal gets noisier, ERM increasingly relies on shortcuts. IRM, however, remains focused on the invariant (though noisy) causal features, avoiding the "catastrophic" performance drops ERM suffers when shortcuts are inverted.
  • The Accuracy-Invariance Trade-off: In realistic settings like ActorCorr, where shortcuts are semantically meaningful words, the authors found a trade-off. High invariance penalties effectively suppress the shortcut but can slightly degrade "clean" accuracy, requiring careful hyperparameter calibration.
Causal graphs for (a) Label-Generated Features, (b) Confounding, and (c) Selection Bias.

Key Results

  • OOD Gains: In synthetic experiments where shortcuts were inverted at test time, IRM outperformed ERM by 6.9 to 58.1 percentage points.
  • Gender Bias Mitigation: On the Bias in Bios task, IRM improved mean OOD accuracy from 43.35% to 78.78% across ten profession pairs. The gains were most pronounced when professions were semantically similar (e.g., attorney/paralegal), where ERM is most prone to relying on gender cues.
  • Actor Name Shortcuts: In ActorCorr, IRM reduced the sensitivity to misleading actor cues (measured by Anti-Correlated Accuracy Change) from 0.186 down to 0.049, though this came with a small drop in overall pooled accuracy.

Limitations and Future Work

The authors acknowledge several limitations:

  • Controlled Shortcuts: While they moved toward realistic settings, the initial causal mechanisms were injected via controlled token injection, which may not capture the full complexity of organic, diffuse shortcuts in natural text.
  • Linear Head Limitation: By freezing the DistilBERT encoder, the study isolates the effect of the IRM objective but doesn't explore how IRM interacts with the high-dimensional, over-parameterized space of a fully fine-tuned Transformer.
  • Environment Definition: IRM's success hinges on having a meaningful partition of data into environments. In many real-world scenarios, defining these boundaries without prior knowledge of the bias remains a challenge.

Future work includes extending this analysis to fully fine-tuned models and exploring ways to discover environment partitions automatically.

Summary updated

Abstract

In NLP, models trained via Empirical Risk Minimization (ERM) often exploit spurious correlations, such as dialectal markers, at the expense of robust linguistic features, thereby degrading their out-of-distribution (OOD) generalization. Invariant Risk Minimization (IRM) aims to suppress such shortcuts by identifying predictive relationships that remain stable across environments, but its practical utility for NLP remains debated. We use a causality-driven framework to evaluate IRM under three shortcut-generating mechanisms: label-generated features, confounding, and selection bias. Our results show that IRM successfully rejects shortcuts across the evaluated causal and anti-causal structures, including under limited environmental variation and increasing label noise. We then extend our analysis beyond controlled synthetic artifacts to two less constrained settings: gender-profession associations in Bias in Bios and semantically loaded actor-name shortcuts in ActorCorr. These experiments show that invariant learning can mitigate more realistic textual shortcuts, while semantically meaningful features require careful calibration of the invariance penalty.
BibTeX
@misc{grunenwald2026disentangling,
  author = {Paul Grunenwald and Benoît Crabbé and Guillaume Wisniewski and Pirmin Lemberger},
  title = {{Disentangling Spurious Correlations: How IRM Handles Diverse Shortcuts in Text Classification}},
  year = {2026},
  eprint = {2611.17107},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17107},
}
Figure