Abstract
A significant challenge in developing Large Language Models (LLMs) is the trade-off between specialized skill acquisition and general linguistic intelligence. Fine-tuning models on non-linguistic tasks, such as arithmetic, often leads to "catastrophic forgetting" of core language capabilities. This paper introduces Skill-LM, a framework designed to learn arithmetic reasoning without sacrificing linguistic proficiency. By employing Elastic Weight Consolidation (EWC) for information-theoretic weight anchoring and a novel magnitude-aware regression loss, Skill-LM outperforms existing models using significantly less data and no synthetic linguistic samples.
Introduction
The integration of quantitative reasoning into Natural Language Processing (Math-NLP) aims to provide LLMs with calculator-like precision. However, traditional fine-tuning on niche tasks often overwrites weights essential for grammar or sentiment analysis. Previous solutions, like GenBERT, relied on computationally expensive "rehearsal" methods—mixing millions of synthetic language samples into training data. Skill-LM addresses the root cause by identifying "load-bearing" linguistic weights and treating numbers as magnitudes rather than mere text tokens.
Methodology
The Skill-LM framework utilizes two primary interventions to maintain a balance between new skills and existing knowledge:
1. Linguistic Retention via Elastic Weight Consolidation (EWC)
The researchers first calculate the Fisher Information Index of the base model on a general corpus (WikiText). This index identifies which specific parameters are critical for linguistic tasks. During arithmetic training, EWC penalizes significant changes to these vital weights, effectively "anchoring" the model to its linguistic foundations.
2. Magnitude-Aware Learning ($L_{REG}$)
Standard Cross-Entropy loss treats numbers as categorical tokens, ignoring their numerical relationships. Skill-LM introduces a Regression Loss ($L_{REG}$) that forces the model to learn actual numerical values. This helps the model understand the "number line," leading to better generalization on out-of-domain (OOD) data.
Key Insights: Task-Specific Forgetting
Analysis using Fisher Information revealed that forgetting is not uniform across all linguistic skills. Arithmetic training overlaps significantly with parameters used for grammatical correctness (CoLA) but shows minimal overlap with those used for pronoun resolution (WNLI). This explains why certain linguistic capabilities are more vulnerable during specialized fine-tuning.
Results
Skill-LM demonstrates remarkable efficiency and robustness:
- Data Efficiency: It matches state-of-the-art performance (GenBERT) using 75% less arithmetic training data and zero synthetic language samples.
- Language Preservation: On the CoLA grammar task, while a naively fine-tuned BERT's score dropped from 0.59 to 0.08, Skill-LM maintained a score of 0.58.
- Mathematical Generalization: In tests involving numbers larger than those in the training set, Skill-LM achieved 19.44% accuracy, compared to 1.32% for GenBERT. Qualitative results showed that Skill-LM's errors were typically within the correct order of magnitude.
Conclusion
Skill-LM provides a scalable path for enhancing LLMs with specialized non-linguistic skills while preserving their general-purpose utility. By focusing on the locality of parameterization and magnitude-aware learning, the framework avoids the pitfalls of catastrophic forgetting without the overhead of massive synthetic data rehearsal.
Summary updated