Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning

Siyu Ren, Kenny Q. Zhu

Introduction

Pre-trained Language Models (PLMs) like BERT have revolutionized NLP, yet their internal mechanisms for storing and retrieving relational knowledge remain opaque. Researchers from Shanghai Jiao Tong University have recently demonstrated that these models contain specialized, "entangled" subnetworks dedicated to specific relational knowledge. By employing network pruning to isolate these latent representation subspaces, they created "specialized" models that significantly outperform their full-scale counterparts in relational reasoning and commonsense tasks, often while utilizing 50% fewer active parameters.

The Challenge of Relational Knowledge

While PLMs inherit vast amounts of relational knowledge (e.g., "a bus has seats") during pre-training, standard fine-tuning often "muddies the waters." By updating all parameters, the model may memorize spurious patterns rather than leveraging its internal logic. This research addresses a fundamental question: Can we extract specific "circuits" or subnetworks within a PLM dedicated to particular relations? If so, can these specialized subnetworks reason more effectively than the original "jack-of-all-trades" model?

Methodology: Pruning as Specialization

The authors treat specialization as a network pruning task. Instead of adding new layers or modifying weights, they identify a binary mask—a set of "on/off" switches for existing weights—that optimizes the model for a specific relation.

Key components of this approach include:

  • Relation-Specific Pruning: Subnetworks are defined by identifying weights most predictive for a relation's "cloze" prompts (e.g., "You are likely to find [MASK] in a bus").
  • Deterministic Pruning: The researchers found that using a hard threshold with a Straight-Through Estimator for gradients was more effective than probabilistic masking.
  • Mask Union: For multi-relation tasks, the model can activate multiple specialized circuits simultaneously by taking the union of relevant masks.
Pruned BERT focuses attention on relevant entities (e.g., 'bus') to correctly predict relational knowledge ('seats').

As shown in the figure above, pruning shifts the model's attention. In the original BERT, the [MASK] token often attends to "noise" like punctuation. In the pruned subnetwork, attention shifts dramatically toward relevant entities, explaining the jump in reasoning accuracy.

Key Insights and Quantitative Results

The study yielded several striking findings regarding knowledge disentanglement and model performance:

  1. Knowledge Disentanglement: Subnetworks are highly specific. Applying a mask intended for one relation (e.g., "CapableOf") to a different one (e.g., "HasA") causes performance to plummet, proving that relational knowledge is disentangled within the architecture.
  2. Massive Accuracy Gains: On the C-LAMA test set, a pruned BERT-base achieved a 43.8% P@1 score, a nearly four-fold increase over the original model's 12.9%.
  3. Efficiency: These gains were achieved at approximately 50% sparsity, meaning the specialized models are effectively half the size of the originals.
Performance comparison (P@K) between the original BERT and pruned subnetworks across different K values.

The quantitative results in the figure above highlight that when the pruning mask matches the relation ("Pruned(match)"), the performance significantly exceeds the original model. Conversely, mismatched or random masks lead to poor performance, reinforcing the "Lottery Ticket" hypothesis for semantic knowledge.

Conclusion and Future Work

This work suggests that PLMs are essentially collections of specialized "expert" subnetworks. Pruning acts as a "cleaner" specialization than fine-tuning, uncovering robust latent structures already present from pre-training.

While successful, the authors note limitations, such as the heuristic nature of mask combination and the current focus on binary commonsense relations. Future research may extend this framework to factual knowledge (e.g., "X was born in Y") and more complex n-ary relations, potentially leading to even more efficient and capable reasoning engines.

Summary updated

Abstract

Pretrained masked language models (PLMs) were shown to be inheriting a considerable amount of relational knowledge from the source corpora. In this paper, we present an in-depth and comprehensive study concerning specializing PLMs into relational models from the perspective of network pruning. We show that it is possible to find subnetworks capable of representing grounded commonsense relations at non-trivial sparsity while being more generalizable than original PLMs in scenarios requiring knowledge of single or multiple commonsense relations.
BibTeX
@inproceedings{ren2022specializing,
  author = {Siyu Ren and Kenny Q. Zhu},
  title = {{Specializing Pre-trained Language Models for Better Relational Reasoning via Network Pruning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-naacl.169},
}
Figure