Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

Daniel Andor, Luheng He, Kenton Lee, Emily Pitler

Abstract

Reading comprehension models have been successfully applied to extractive text answers, but it is unclear how best to generalize these models to abstractive numerical answers. We enable a BERT-based reading comprehension model to perform lightweight numerical reasoning. We augment the model with a predefined set of executable 'programs' which encompass simple arithmetic as well as extraction. Rather than having to learn to manipulate numbers directly, the model can pick a program and execute it. On the recent Discrete Reasoning Over Passages (DROP) dataset, designed to challenge reading comprehension models, we show a 33% absolute improvement by adding shallow programs. The model can learn to predict new operations when appropriate in a math word problem setting (Roy and Roth, 2015) with very few training examples.

Introduction

Standard Transformer models like BERT have revolutionized extractive Question Answering (QA), but they remain "literate but innumerate." While they excel at identifying relevant text spans, they struggle with basic arithmetic. If a question requires calculating a value not explicitly stated—such as "How many more goals did Team A score than Team B?"—standard models often fail because they lack discrete reasoning capabilities.

This paper introduces a hybrid approach that augments BERT with a symbolic "calculator," allowing the model to predict executable programs (e.g., Diff(num1, num2)) rather than just text spans. This architectural shift leads to a massive 33% absolute improvement on the DROP reasoning benchmark.

Methodology: Teaching BERT to Use a Calculator

The core innovation is treating numerical reasoning as a semantic parsing problem. Instead of forcing the neural network to perform arithmetic internally, the authors train it to select and parameterize symbolic programs from a predefined library.

Program Synthesis and Operation Library

The model is modified to predict a "derivation" in the format Operation(arguments). The operation library includes arithmetic functions like Sum, Diff, and Sum3, as well as specialized operations like Diff100 for percentage-based reasoning. It also retains standard extraction capabilities through Span and Merge operations.

Table 1: The model's library of operations and example derivations.

Argument Selection and Scoring

Arguments for these operations are pointers to specific numbers or text spans identified within the passage. The system uses BERT to generate token representations, which are then processed by a Multi-Layer Perceptron (MLP) to score potential programs and their arguments. The highest-scoring program is executed by a deterministic calculator to produce the final answer.

Experimental Evaluation

The primary benchmark for this work is the DROP (Discrete Reasoning Over Passages) dataset, specifically designed to challenge models with questions requiring addition, subtraction, counting, and sorting. To maintain general reading comprehension abilities, the model is co-trained on the CoQA dataset. Additionally, the Illinois Math Word Problems dataset is used to test the model's ability to learn new operations, such as multiplication and division, in a few-shot setting.

Key Results

The augmented model achieved a 33% absolute improvement in Exact Match (EM) over the previous state-of-the-art (NAQANet) on the DROP dataset. The ensemble version reached 78.97 EM and 82.56 F1 on the development set, significantly closing the gap toward human performance.

Table 2: Performance on the DROP dataset compared to the NAQANet baseline.

Insights

  • Interpretability: Because the model outputs a symbolic program, its reasoning process is transparent. One can see exactly which numbers were chosen and why a specific operation was applied.
  • Cross-Domain Transfer: The model demonstrated an ability to learn new operations (multiplication/division) with very few examples on the Illinois dataset, suggesting that "learning to find arguments" is a generalizable skill.

Discussion and Limitations

While successful, the system is currently limited to "shallow" programs, typically involving only one or two steps. It cannot yet synthesize the complex, multi-step nested formulas required for more sophisticated word problems. Additionally, the search space for programs grows rapidly with the number of operations and arguments, requiring pruning strategies. Finally, the system relies on a heuristic pre-processor to identify numbers; if a number is missed during pre-processing, the model cannot use it as an argument. Future work will focus on more complex compositional answers and improved strategies for searching the program space.

Summary updated

Abstract

Reading comprehension models have been successfully applied to extractive text answers, but it is unclear how best to generalize these models to abstractive numerical answers. We enable a BERT-based reading comprehension model to perform lightweight numerical reasoning. We augment the model with a predefined set of executable 'programs' which encompass simple arithmetic as well as extraction. Rather than having to learn to manipulate numbers directly, the model can pick a program and execute it. On the recent Discrete Reasoning Over Passages (DROP) dataset, designed to challenge reading comprehension models, we show a 33% absolute improvement by adding shallow programs. The model can learn to predict new operations when appropriate in a math word problem setting (Roy and Roth, 2015) with very few training examples.
BibTeX
@inproceedings{andor2019giving,
  author = {Daniel Andor and Luheng He and Kenton Lee and Emily Pitler},
  title = {{Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=VX1ZgbExu93},
}
Figure