The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models

Adithya Bhaskar, Dan Friedman, Danqi Chen

Abstract

Even within a single fine-tuned language model, different subnetworks can achieve identical in-domain performance while generalizing vastly differently to out-of-distribution (OOD) data. This research identifies a "heuristic core"—a shared set of attention heads that compute simple shortcut features. The findings suggest that generalization is not a competition between disjoint algorithms, but rather a process of building complex reasoning on top of these foundational heuristics.

Introduction

Pretrained language models (LMs) are notorious for taking "shortcuts." For example, in Natural Language Inference (NLI), a model might learn a "lexical overlap" heuristic: if two sentences share many words, it predicts "entailment," even if the meaning is different (e.g., "The doctor saw the lawyer" vs. "The lawyer saw the doctor").

Previous research showed that different training runs (different random seeds) result in models with similar in-domain accuracy but wildly different OOD generalization. This paper investigates whether the model contains multiple "competing subnetworks" representing different algorithms or if a more integrated mechanism is at play. Understanding this is crucial for building robust models that do not fail when encountering adversarial or slightly shifted data distributions.

Methods

The authors utilize structured pruning to analyze BERT, RoBERTa, and GPT-2.

  • Subnetwork Identification: Using $L_0$ regularization and "binary mask" optimization (specifically the CoFi pruning recipe), they identify subsets of attention heads and MLP layers that approximate the full model's behavior.
  • Effective Size: This is defined as the smallest subnetwork that remains "faithful" (within 3% accuracy) to the full model on a specific dataset.
  • Ablation Studies: To understand specific components, individual attention heads are systematically disabled (ablated) to measure performance drops across in-domain and OOD tasks.

The study focuses on sentence-pair classification tasks using MNLI (Multi-Genre Natural Language Inference) and QQP (Quora Question Pairs) for in-domain training, with HANS and PAWS-QQP serving as adversarial OOD evaluation sets.

The Heuristic Core

Instead of finding disjoint subnetworks that "compete" for dominance, the authors discovered a "heuristic core." In BERT-MNLI, this consists of 9 attention heads that appear in every subnetwork, regardless of its generalization capability. These heads emerge early in training and compute "shallow" features, such as identifying repeated words.

Conceptual comparison between the competing subnetworks hypothesis and the discovered heuristic core architecture.

Key insights regarding this architecture include:

  • Heuristics as Foundations: When the "heuristic core" heads are ablated, OOD performance plummets significantly more than in-domain performance. This suggests that the "smart" parts of the model rely on the outputs of these "dumb" heuristic heads to perform complex reasoning.
  • Effective Size and Generalization: In simple tasks, generalization often leads to smaller effective model sizes. In LMs, the opposite occurs: effective size increases when the model generalizes, suggesting that generalizing components are built on top of the heuristic core.

Key Results

Empirical results demonstrate that at 50% sparsity, different subnetworks vary wildly in OOD performance despite identical in-domain accuracy. For instance, accuracy on the "Embed-If" subcase of HANS ranged from 4.1% to 50.2% across different seeds.

In-domain vs. out-of-distribution accuracy across different subnetworks and sparsity levels for MNLI and QQP tasks.

Further findings include:

  • Super-additive Ablation: Ablating just three specific core heads (L2.H5, L3.H7, and L4.H0) caused OOD accuracy on the "SO-Swap" task to drop from 86.7% to 3.6%, while in-domain accuracy only dropped by approximately 3%.
  • Sparsity Constraints: While models can be pruned by 50% and maintain some generalization, very sparse models (70%+) almost always revert to pure heuristic behavior, even if in-domain accuracy remains respectable.

Conclusion

The discovery of the heuristic core suggests that LMs do not choose between "good" and "bad" algorithms. Instead, they all share a common set of heuristic-driven components, and generalization is achieved by successfully building more sophisticated logic upon that base. Future work is needed to determine if this pattern holds for massive Large Language Models (LLMs) and generative tasks beyond simple classification.

Summary updated

Abstract

Prior work has found that pretrained language models (LMs) fine-tuned with different random seeds can achieve similar in-domain performance but generalize differently on tests of syntactic generalization. In this work, we show that, even within a single model, we can find multiple subnetworks that perform similarly in-domain, but generalize vastly differently. To better understand these phenomena, we investigate if they can be understood in terms of “competing subnetworks”: the model initially represents a variety of distinct algorithms, corresponding to different subnetworks, and generalization occurs when it ultimately converges to one. This explanation has been used to account for generalization in simple algorithmic tasks (“grokking”). Instead of finding competing subnetworks, we find that all subnetworks—whether they generalize or not—share a set of attention heads, which we refer to as the _heuristic core_. Further analysis suggests that these attention heads emerge early in training and compute shallow, non-generalizing features. The model learns to generalize by incorporating additional attention heads, which depend on the outputs of the “heuristic” heads to compute higher-level features. Overall, our results offer a more detailed picture of the mechanisms for syntactic generalization in pre-trained LMs.
BibTeX
@inproceedings{bhaskar2024heuristic,
  author = {Adithya Bhaskar and Dan Friedman and Danqi Chen},
  title = {{The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=mKSbr8yEJe},
}
Figure