Abstract
Neural NLP models frequently exhibit "superhuman" performance on standard benchmarks like GLUE, yet they remain remarkably brittle when encountering data that deviates from their training distribution. This survey provides a unifying framework for three previously fragmented research areas: adversarial robustness, domain generalization, and dataset biases. By categorizing these challenges as forms of "covariate shift," the authors map the landscape of out-of-distribution (OOD) data generation and evaluation, revealing critical trade-offs where progress in one area may inadvertently hinder performance in others.
Introduction: The Brittleness of Neural NLP
Despite the rapid advancement of Large Language Models (LLMs), their reliability in real-world scenarios remains a concern. Historically, the research community has addressed this "brittleness" through three separate lenses:
- Adversarial Robustness: Focusing on defending against intentional, malicious perturbations designed to flip model predictions.
- Domain Generalization: Ensuring models can transition from one "genre" (e.g., news articles) to another (e.g., social media posts) without significant performance drops.
- Dataset Biases: Preventing models from relying on "shortcuts" or spurious correlations—such as assuming the presence of the word "not" always indicates a "contradiction" label in Natural Language Inference (NLI) tasks.
The lack of an integrated discussion between these fields has hindered the development of truly robust models. A technique that "debiases" a model might inadvertently make it more vulnerable to adversarial attacks or less capable of generalizing across domains.
A Unifying Taxonomy of Distribution Shift
The core contribution of this survey is a taxonomy based on the concept of Distribution Shift. The authors categorize input features into two types:
- Semantic Features ($s_{rel}$): Information directly relevant to the task (e.g., the core sentiment of a sentence).
- Background Features ($s_{ir}$): Task-irrelevant information that may still be present in the input (e.g., writing style, syntax, or specific keywords).
Biased Features are background features that a model mistakenly learns as predictive. Using this lens, OOD evaluation is categorized into:
- Natural Domain Shift (NDS): Shifts in both background and semantic features across different genres or demographics.
- Debiased Evaluation: Removing biased features to test if the model understands the underlying task.
- Adversarial Evaluation: Intentionally shifting features to "trick" the model.
The Landscape of OOD Evaluation
The survey highlights significant differences in how OOD data is generated and evaluated across these three research lines.
As shown in the table above, while Domain Generalization typically relies on existing "natural" data, Adversarial and Debiasing research often requires automated or human-in-the-loop generation to create challenging, "unnatural" examples. Furthermore, a mismatch exists in evaluation metrics: domain generalization and debiasing use standard accuracy (ID metrics), whereas adversarial research focuses on Attack Success Rate (ASR).
Adversarial Perturbations and Model Knowledge
Adversarial attacks are classified by the level of knowledge an attacker has about the target model.
White-box attacks, which leverage the model's gradients, are highly effective at identifying and exploiting biased features. In contrast, Black-box attacks often rely on paraphrasing or character editing to shift background styles without needing internal model access.
Key Insights: The "Shortcut" Trap and Conflicting Goals
Models are "lazy" learners, often prioritizing background features (shortcuts) over complex semantic features because they are easier to optimize during training on biased datasets.
A critical finding of the survey is that OOD robustness goals can be conflicting. For instance, Systematic Debiasing—filtering a dataset to remove shortcuts—can lead to decreased adversarial robustness. This occurs because the model loses the "knowledge" of background features it might need to distinguish a legitimate input from a noisy adversarial perturbation.
Future Directions
The authors call for several shifts in NLP research:
- Unified Benchmarking: The community needs benchmarks that evaluate all three OOD types simultaneously to prevent over-optimization for one type of robustness at the expense of others.
- Label Distribution Shifts: Most current research assumes the relationship between input and label ($P(Y|X)$) is fixed. Future work must address scenarios where labels are ambiguous or change based on context.
- Realistic Harm: Research should move beyond "academic" attacks (like changing a single character) toward addressing real-world harms, such as the generation of toxic content or misinformation.
By bridging the gaps between adversarial robustness, domain generalization, and dataset biases, this survey provides a roadmap for building NLP models that are not just accurate on benchmarks, but truly reliable in the wild.
Summary updated