Injecting Numerical Reasoning Skills into Language Models

Mor Geva, Ankit Gupta, Jonathan Berant

Abstract

Standard language models often struggle with tasks requiring numerical reasoning, such as arithmetic or quantitative comparisons. This paper introduces GENBERT, a model designed to overcome these limitations by injecting numerical skills through large-scale synthetic pre-training. By training on millions of automatically generated math problems and textual word problems, GENBERT achieves state-of-the-art performance on the DROP benchmark and demonstrates significant zero-shot generalization to math word problems, all while maintaining its core linguistic capabilities.

Introduction

Large language models (LMs) like BERT excel at natural language understanding but are notoriously "math-illiterate." Their primary training objective—predicting masked words—does not inherently teach the logic of arithmetic. Previous attempts to address this have often relied on hybrid architectures with hard-coded symbolic modules. However, these modules are often rigid and task-specific.

The authors propose a different philosophy: numerical reasoning should be treated as a skill to be injected into a general-purpose model via data. This approach avoids architectural complexity and allows the model to learn the flexible mapping between natural language and numerical operations.

Methodology

The researchers developed GENBERT, an encoder-decoder architecture based on BERT. Unlike purely extractive models, GENBERT's generative decoder allows it to produce numerical values that may not appear in the input context.

Skill Injection via Synthetic Data

The core of the methodology is a two-stage pre-training process using synthetic data:

  1. Numerical Data (ND): The model is first exposed to millions of simple arithmetic expressions (e.g., 517.4 - 17484 + 1013.21). This stage focuses on teaching the underlying logic of addition, subtraction, and decimal handling.
  2. Textual Data (TD): The model then trains on synthetic math word problems generated from a template grammar. This teaches the model to translate natural language descriptions into numerical reasoning steps.
Overview of the GENBERT pre-training and fine-tuning pipeline.

Technical Innovations

To facilitate learning, the authors implemented Digit Tokenization (DT), where numbers are split into individual digits (e.g., "123" becomes "1", "2", "3"). This helps the model grasp place-value logic more effectively than standard subword tokenization. Additionally, Multi-task Training with the standard Masked Language Modeling (MLM) objective was used throughout to prevent "catastrophic forgetting" of general language skills.

Experimental Setup

The model was evaluated on several key benchmarks:

  • DROP (Discrete Reasoning Over Paragraphs): A challenging dataset requiring complex reasoning and math over Wikipedia text.
  • Math Word Problems (MWP): Used for zero-shot evaluation to test the generalization of learned math skills.
  • SQuAD: A standard reading comprehension benchmark used to monitor general linguistic performance.

Results

GENBERT demonstrated significant improvements across the board:

  • DROP Performance: After synthetic pre-training, GENBERT's $F_1$ score on DROP rose from 49.3 to 72.3, matching the performance of much more complex, specialized architectures.
  • Zero-Shot Generalization: On the ADDSUB dataset, GENBERT achieved 22.8% accuracy zero-shot, a massive improvement over the 2% scored by the base model.
  • Linguistic Stability: On SQuAD, the model maintained an $F_1$ of 88.6, identical to the original BERT, confirming that numerical skill injection does not degrade general language understanding.

Discussion and Limitations

The study highlights that numeracy is largely a data problem. By providing sufficient synthetic examples, a transformer can learn the "rules" of math internally without external calculators. However, several challenges remain.

Model performance (EM) as a function of the number of numerical terms in the problem.

As shown in the figure above, there is a clear "complexity ceiling." While the model handles two-term expressions well, performance drops significantly as the number of terms increases to three, and it struggles with four or more terms. Additionally, the current model does not yet handle operations like multiplication, division, or complex sorting tasks effectively.

Conclusion

GENBERT proves that numerical reasoning can be successfully "injected" into language models through targeted synthetic pre-training. This data-driven approach offers a flexible alternative to hard-coded modules. Future work will explore extending this "skill injection" recipe to other domains, such as logical and temporal reasoning.

Summary updated

Abstract

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 $\rightarrow$ 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.
BibTeX
@inproceedings{geva2020injecting,
  author = {Mor Geva and Ankit Gupta and Jonathan Berant},
  title = {{Injecting Numerical Reasoning Skills into Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.89},
}
Figure