Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic

Mandar Sharma, Nikhil Muralidhar, Naren RamakrishnanarXiv 2022 · DOI · arXiv

Introduction: The Challenge of Catastrophic Forgetting

Large language models (LLMs) like GPT and BERT have revolutionized natural language processing, yet they often struggle with "strict" mathematical reasoning and the precise handling of numerals. While fine-tuning on specialized datasets seems like a straightforward solution, it frequently triggers catastrophic forgetting. In this phenomenon, the weights essential for a model's original task (e.g., language understanding) are overwritten to accommodate a new task (e.g., arithmetic), leading to a total loss of the original skill.

This paper investigates this "math vs. language" trade-off using a DistilBERT model. The authors observed that standard fine-tuning on arithmetic tasks caused the model to completely lose its ability to judge grammatical correctness, even as it became proficient in math. To address this, they propose an information-theoretic approach to inject new skills while preserving existing ones.

Methodology: Information-Theoretic Skill Injection

The researchers developed a two-pronged strategy to overcome the barriers of skill injection:

1. Information-Theoretic Analysis

Using Fisher Information, the authors identified which specific parameters in the model are most vital for linguistic tasks. By calculating the sensitivity of parameters, they discovered that linguistic and arithmetic tasks effectively "live" in different regions of the model's parameter space. This sensitivity analysis allows for the identification of a subset of weights that must be protected to maintain linguistic performance.

Fisher Information Sensitivity for Layer 1 and Layer 5, showing the importance of specific parameters for MRPC and CoLA tasks.

2. Elastic Weight Consolidation (EWC)

To operationalize these insights, the authors implemented Elastic Weight Consolidation (EWC). Unlike standard fine-tuning, EWC introduces a penalty to the training loss function. If the optimization process attempts to significantly alter a weight flagged by the Fisher Information matrix as important for language, the penalty increases. This constraint forces the model to find a mathematical solution within the "neighborhood" of its original linguistic capabilities, effectively finding a "sweet spot" in the parameter space.

Visualizing the Parameter Space

To validate their approach, the authors employed t-SNE projections to map the high-dimensional parameter space. These visualizations confirmed that models trained solely on arithmetic drift significantly away from the clusters formed by models trained on linguistic tasks.

t-SNE projections of the encoder parameter space for Self-attention Layer-1 and Layer-5, illustrating the separation between arithmetic and linguistic task regions.

Key Results and Discussion

The effectiveness of the EWC-based approach was evaluated against a base DistilBERT and a standard math-tuned version across several benchmarks:

  • Mathematical Proficiency: The EWC model achieved the highest arithmetic accuracy, with a log Root Mean Square Error (ln RMSE) of 0.4360, a vast improvement over the base model's 3.5367.
  • Linguistic Preservation: The most striking result was in grammatical correctness (CoLA). While standard math-tuning caused the CoLA score to plummet from 0.4827 to 0.0000, the EWC model maintained a score of 0.4193.
  • Task Consistency: Across other GLUE tasks like MNLI (logic) and SST-2 (sentiment), the EWC model remained within a few percentage points of the base model, whereas standard fine-tuning led to significant degradation.

Conclusion and Future Directions

This research demonstrates that the trade-off between mathematical and linguistic skills in LLMs is not inevitable. By using information-theoretic constraints like EWC, it is possible to inject specialized skills into a model while preserving its foundational capabilities.

Future work will look to expand this framework beyond simple addition and subtraction to include more complex operations like multiplication and division, as well as handling decimal-point numerals. Ultimately, the authors aim to apply this methodology to a broader range of non-linguistic skills, paving the way for more versatile and robust multi-task language models.

Summary updated

Abstract

Through their transfer learning abilities, highly-parameterized large pre-trained language models have dominated the NLP landscape for a multitude of downstream language tasks. Though linguistically proficient, the inability of these models to incorporate the learning of non-linguistic entities (numerals and arithmetic reasoning) limits their usage for tasks that require numeric comprehension or strict mathematical reasoning. However, as we illustrate in this paper, building a general purpose language model that also happens to be proficient in mathematical reasoning is not as straight-forward as training it on a numeric dataset. In this work, we develop a novel framework that enables language models to be mathematically proficient while retaining their linguistic prowess. Specifically, we offer information-theoretic interventions to overcome the catastrophic forgetting of linguistic skills that occurs while injecting non-linguistic skills into language models.
BibTeX
@misc{sharma2022overcoming,
  author = {Mandar Sharma and Nikhil Muralidhar and Naren Ramakrishnan},
  title = {{Overcoming Barriers to Skill Injection in Language Modeling: Case Study in Arithmetic}},
  year = {2022},
  eprint = {2211.02098},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.02098},
}
Figure