Abstract
Even within a single fine-tuned language model, different subnetworks can achieve identical in-domain performance while generalizing vastly differently to out-of-distribution (OOD) data. This research identifies a "heuristic core"—a shared set of attention heads that compute simple shortcut features. The findings suggest that generalization is not a competition between disjoint algorithms, but rather a process of building complex reasoning on top of these foundational heuristics.
Introduction
Pretrained language models (LMs) are notorious for taking "shortcuts." For example, in Natural Language Inference (NLI), a model might learn a "lexical overlap" heuristic: if two sentences share many words, it predicts "entailment," even if the meaning is different (e.g., "The doctor saw the lawyer" vs. "The lawyer saw the doctor").
Previous research showed that different training runs (different random seeds) result in models with similar in-domain accuracy but wildly different OOD generalization. This paper investigates whether the model contains multiple "competing subnetworks" representing different algorithms or if a more integrated mechanism is at play. Understanding this is crucial for building robust models that do not fail when encountering adversarial or slightly shifted data distributions.
Methods
The authors utilize structured pruning to analyze BERT, RoBERTa, and GPT-2.
- Subnetwork Identification: Using $L_0$ regularization and "binary mask" optimization (specifically the CoFi pruning recipe), they identify subsets of attention heads and MLP layers that approximate the full model's behavior.
- Effective Size: This is defined as the smallest subnetwork that remains "faithful" (within 3% accuracy) to the full model on a specific dataset.
- Ablation Studies: To understand specific components, individual attention heads are systematically disabled (ablated) to measure performance drops across in-domain and OOD tasks.
The study focuses on sentence-pair classification tasks using MNLI (Multi-Genre Natural Language Inference) and QQP (Quora Question Pairs) for in-domain training, with HANS and PAWS-QQP serving as adversarial OOD evaluation sets.
The Heuristic Core
Instead of finding disjoint subnetworks that "compete" for dominance, the authors discovered a "heuristic core." In BERT-MNLI, this consists of 9 attention heads that appear in every subnetwork, regardless of its generalization capability. These heads emerge early in training and compute "shallow" features, such as identifying repeated words.
Key insights regarding this architecture include:
- Heuristics as Foundations: When the "heuristic core" heads are ablated, OOD performance plummets significantly more than in-domain performance. This suggests that the "smart" parts of the model rely on the outputs of these "dumb" heuristic heads to perform complex reasoning.
- Effective Size and Generalization: In simple tasks, generalization often leads to smaller effective model sizes. In LMs, the opposite occurs: effective size increases when the model generalizes, suggesting that generalizing components are built on top of the heuristic core.
Key Results
Empirical results demonstrate that at 50% sparsity, different subnetworks vary wildly in OOD performance despite identical in-domain accuracy. For instance, accuracy on the "Embed-If" subcase of HANS ranged from 4.1% to 50.2% across different seeds.
Further findings include:
- Super-additive Ablation: Ablating just three specific core heads (L2.H5, L3.H7, and L4.H0) caused OOD accuracy on the "SO-Swap" task to drop from 86.7% to 3.6%, while in-domain accuracy only dropped by approximately 3%.
- Sparsity Constraints: While models can be pruned by 50% and maintain some generalization, very sparse models (70%+) almost always revert to pure heuristic behavior, even if in-domain accuracy remains respectable.
Conclusion
The discovery of the heuristic core suggests that LMs do not choose between "good" and "bad" algorithms. Instead, they all share a common set of heuristic-driven components, and generalization is achieved by successfully building more sophisticated logic upon that base. Future work is needed to determine if this pattern holds for massive Large Language Models (LLMs) and generative tasks beyond simple classification.
Summary updated