Abstract
Standard language models often struggle with tasks requiring numerical reasoning, such as arithmetic or quantitative comparisons. This paper introduces GENBERT, a model designed to overcome these limitations by injecting numerical skills through large-scale synthetic pre-training. By training on millions of automatically generated math problems and textual word problems, GENBERT achieves state-of-the-art performance on the DROP benchmark and demonstrates significant zero-shot generalization to math word problems, all while maintaining its core linguistic capabilities.
Introduction
Large language models (LMs) like BERT excel at natural language understanding but are notoriously "math-illiterate." Their primary training objective—predicting masked words—does not inherently teach the logic of arithmetic. Previous attempts to address this have often relied on hybrid architectures with hard-coded symbolic modules. However, these modules are often rigid and task-specific.
The authors propose a different philosophy: numerical reasoning should be treated as a skill to be injected into a general-purpose model via data. This approach avoids architectural complexity and allows the model to learn the flexible mapping between natural language and numerical operations.
Methodology
The researchers developed GENBERT, an encoder-decoder architecture based on BERT. Unlike purely extractive models, GENBERT's generative decoder allows it to produce numerical values that may not appear in the input context.
Skill Injection via Synthetic Data
The core of the methodology is a two-stage pre-training process using synthetic data:
- Numerical Data (ND): The model is first exposed to millions of simple arithmetic expressions (e.g.,
517.4 - 17484 + 1013.21). This stage focuses on teaching the underlying logic of addition, subtraction, and decimal handling. - Textual Data (TD): The model then trains on synthetic math word problems generated from a template grammar. This teaches the model to translate natural language descriptions into numerical reasoning steps.
Technical Innovations
To facilitate learning, the authors implemented Digit Tokenization (DT), where numbers are split into individual digits (e.g., "123" becomes "1", "2", "3"). This helps the model grasp place-value logic more effectively than standard subword tokenization. Additionally, Multi-task Training with the standard Masked Language Modeling (MLM) objective was used throughout to prevent "catastrophic forgetting" of general language skills.
Experimental Setup
The model was evaluated on several key benchmarks:
- DROP (Discrete Reasoning Over Paragraphs): A challenging dataset requiring complex reasoning and math over Wikipedia text.
- Math Word Problems (MWP): Used for zero-shot evaluation to test the generalization of learned math skills.
- SQuAD: A standard reading comprehension benchmark used to monitor general linguistic performance.
Results
GENBERT demonstrated significant improvements across the board:
- DROP Performance: After synthetic pre-training, GENBERT's $F_1$ score on DROP rose from 49.3 to 72.3, matching the performance of much more complex, specialized architectures.
- Zero-Shot Generalization: On the ADDSUB dataset, GENBERT achieved 22.8% accuracy zero-shot, a massive improvement over the 2% scored by the base model.
- Linguistic Stability: On SQuAD, the model maintained an $F_1$ of 88.6, identical to the original BERT, confirming that numerical skill injection does not degrade general language understanding.
Discussion and Limitations
The study highlights that numeracy is largely a data problem. By providing sufficient synthetic examples, a transformer can learn the "rules" of math internally without external calculators. However, several challenges remain.
As shown in the figure above, there is a clear "complexity ceiling." While the model handles two-term expressions well, performance drops significantly as the number of terms increases to three, and it struggles with four or more terms. Additionally, the current model does not yet handle operations like multiplication, division, or complex sorting tasks effectively.
Conclusion
GENBERT proves that numerical reasoning can be successfully "injected" into language models through targeted synthetic pre-training. This data-driven approach offers a flexible alternative to hard-coded modules. Future work will explore extending this "skill injection" recipe to other domains, such as logical and temporal reasoning.
Summary updated