Abstract
Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 $\rightarrow$ 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.
Introduction
While large pre-trained language models (LMs) like BERT excel at linguistic tasks, they frequently struggle with numerical reasoning. Standard training objectives, such as predicting masked tokens, do not explicitly teach the arithmetic logic required to solve math problems embedded in text. Previous solutions often involved "specialized architectures"—rigid modules like external calculators or hard-coded counting mechanisms.
This paper introduces GenBERT, a model that treats numerical reasoning as a learnable skill rather than a hardware requirement. By leveraging large-scale synthetic data, the authors demonstrate that numerical reasoning can be "injected" into a general-purpose encoder-decoder architecture without sacrificing original linguistic capabilities.
Methodology
The core of the approach is a multi-task pre-training "recipe" designed to teach the model arithmetic through synthetic examples.
Architecture and Tokenization
GenBERT utilizes an encoder-decoder version of BERT. The encoder processes the input text, while the decoder generates the answer token-by-token. This flexibility allows the model to either extract spans from the text or generate new numerical values. To facilitate arithmetic learning, the authors implemented Digit Tokenization (DT), which splits numbers (e.g., "123") into individual digits ("1", "2", "3"), allowing the model to learn the underlying logic of place value and operations.
Synthetic Data Injection
The model undergoes a pre-training phase using two primary synthetic datasets:
- Numerical Data (ND): 1 million examples of pure mathematical expressions (e.g., addition and subtraction of decimals).
- Textual Data (TD): 2.5 million synthetic word problems generated from templates to bridge the gap between language and math.
To prevent "catastrophic forgetting," where the model loses its linguistic knowledge while learning math, training continues on standard Masked Language Modeling (MLM) tasks alongside these new datasets.
Results and Insights
GenBERT was evaluated on several benchmarks, including DROP (Discrete Reasoning Over Paragraphs), MAWPS (math word problems), and SQuAD (reading comprehension).
- Performance Gains: On the DROP benchmark, GenBERT's performance increased from 49.3 to 72.3 F1 after numerical injection, matching state-of-the-art specialized models.
- Generalization: The model showed significant zero-shot improvements on math word problem datasets (like ADDSUB), indicating that the learned skills are transferable beyond the synthetic training distribution.
- Linguistic Integrity: Performance on SQuAD remained stable (81.3 EM vs 81.1 for base BERT), proving that new skills can be added without degrading existing ones.
Limitations
Despite its success, GenBERT faces a "complexity ceiling." Performance degrades significantly as the number of terms in a mathematical expression increases, as shown in the evaluation of multi-term operations.
Additionally, the current implementation focuses primarily on addition and subtraction within specific numerical ranges. Future work could expand synthetic templates to include more complex operations like multiplication, division, and sorting.
Conclusion
GenBERT demonstrates a successful paradigm for skill injection in LMs. By treating reasoning as a data-driven skill, the authors provide a scalable path toward more capable and flexible language models.
Summary updated