Abstract
Large Language Models (LLMs) often lose their core linguistic capabilities when fine-tuned on specialized non-linguistic tasks, a phenomenon known as "catastrophic forgetting." This paper introduces Skill-LM, a framework designed to inject arithmetic skills into LLMs while preserving natural language understanding (NLU). By combining a magnitude-aware regression loss with information-theoretic constraints via Elastic Weight Consolidation (EWC), Skill-LM achieves state-of-the-art performance in arithmetic reasoning—particularly in out-of-domain scenarios—while maintaining linguistic integrity using 75% less training data than previous methods.
Introduction
While modern LLMs demonstrate remarkable fluency, they frequently struggle with the underlying logic of numeracy. When researchers attempt to bridge this gap by fine-tuning models on mathematical datasets, they often encounter a zero-sum game: as the model gains mathematical proficiency, its performance on standard linguistic benchmarks (such as grammar and sentiment analysis) plummets. This "catastrophic forgetting" occurs because the parameter shifts required to master arithmetic logic often overwrite the delicate configurations necessary for natural language processing.
Through Fisher Information analysis, this research identifies a specific parameter conflict. Tasks such as judging grammatical correctness (e.g., the CoLA benchmark) and performing arithmetic reasoning compete for the same high-importance parameters within the model. Skill-LM was developed to navigate this conflict, seeking a "sweet spot" in the parameter space where a model can function effectively as both a calculator and a linguist.
The Skill-LM Framework
The Skill-LM framework introduces two primary innovations to mitigate forgetting while enhancing numerical reasoning:
1. Multi-task Skill-Injection Loss ($L_Q$)
Standard LLMs typically treat numbers as arbitrary tokens using Cross-Entropy loss. Skill-LM supplements this with a Regression Loss ($L_{REG}$). This forces the model to internalize the actual magnitude of numerals. By penalizing the model based on the distance between its prediction and the true value (e.g., a guess of 90 for an answer of 100 is penalized less than a guess of 5), the model learns the "scale" of numbers rather than just their sequence. This acts as a powerful regularizer, improving the model's ability to generalize to numbers larger than those seen during training.
2. Elastic Weight Consolidation (EWC)
To protect linguistic knowledge, the authors employ an information-theoretic approach. Using the Fisher Information index calculated on a proxy dataset (WikiText), the framework identifies which specific parameters are most critical for language tasks. During arithmetic training, a quadratic penalty is applied to these vital parameters. This penalty acts like a mechanical spring, resisting changes to linguistic "neurons" while allowing less critical parameters to adapt to the new mathematical tasks.
Experimental Design
The framework was evaluated using three primary data sources:
- Arithmetic Reasoning: A dataset of 165,000 samples involving complex arithmetic (e.g., "61176.23 - 46741.95 = ?").
- GLUE Benchmark: The industry standard for Natural Language Understanding, including tasks like SST-2 (sentiment) and CoLA (linguistic acceptability).
- WikiText: Utilized to establish the baseline parameter importance for the Fisher Information index.
Results and Analysis
Skill-LM demonstrates that it is possible to achieve high-level numeracy without degrading the model's original utility.
Linguistic Retention
In standard fine-tuning scenarios, a BERT-base model's CoLA score (measuring grammatical acceptability) typically drops from 0.59 to 0.08—essentially random performance. In contrast, Skill-LM maintained a score of 0.58, representing near-perfect retention of linguistic proficiency.
Mathematical Accuracy and Generalization
Skill-LM achieved 98.01% accuracy on in-domain arithmetic, matching the performance of previous state-of-the-art models like GenBERT. However, Skill-LM proved significantly more efficient, requiring only one-fourth of the training data and no additional synthetic linguistic data.
Most notably, Skill-LM excelled in Out-of-Domain (OOD) generalization. When tested on numbers much larger than those in the training set, Skill-LM achieved 19.44% accuracy, compared to GenBERT’s 1.32%. Qualitative analysis revealed that even when Skill-LM failed to predict the exact number, it consistently predicted the correct order of magnitude, whereas baseline models failed entirely.
Conclusion and Future Work
Skill-LM provides a robust methodology for expanding the functional range of LLMs without sacrificing their core strengths. While this study focused on BERT-base (110M parameters) and arithmetic, the framework's reliance on information-theoretic constraints suggests it could be scaled to larger models and diverse non-linguistic skills such as symbolic logic or programming. Future research will focus on applying these techniques to massive-scale models and exploring the impact of linguistic morphology on the forgetting process.
Summary updated