Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension

Daniel Andor, Luheng He, Kenton Lee, Emily Pitler

Abstract: Daniel Andor, Luheng He, Kenton Lee, Emily Pitler. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

While BERT and similar Transformer-based models excel at identifying answers hidden within text, they famously struggle with basic arithmetic. Standard Reading Comprehension (RC) models are primarily "extractive," meaning they are designed to identify a specific span of text that contains the answer. This approach works for benchmarks like SQuAD but fails on the DROP (Discrete Reasoning Over Passages) dataset, where answers often require calculation.

For instance, if a passage mentions 50,000 people from Europe and 30,000 from Asia, and a question asks for the difference, the answer (20,000) is not present in the text. Previous attempts to solve this involved "sign prediction" (assigning a plus or minus to every number in the text), which is difficult to scale and often leads to nonsensical results, such as predicting a negative number of people.

Methods: Neural Orchestration, Symbolic Execution

The authors propose a "program-based" extension to BERT. Instead of just predicting start and end indices for a text span, the model learns to select a derivation—a small executable program—from a predefined set. This hybrid approach uses BERT to create vector representations of the question and passage to identify "pointers" to numbers in the text, while the actual calculation is offloaded to a symbolic calculator tool.

Table of derivations including literals, numerical operations, text spans, and compositions used by the model.

As shown in the table above, the model can select from several derivation types:

  • Literals: Returning fixed values like "Yes", "No", or numbers 0-9.
  • Numerical Operations: Simple arithmetic like Sum(n1, n2) or Diff(n1, n2).
  • Text Spans: Traditional extractive answering.
  • Compositions: More complex operations like Merge(s1, s2) for multi-span answers or Sum3(n1, n2, n3).

By letting BERT "point" to arguments for a calculator, the system becomes more accurate and more interpretable. Because the model outputs a program (e.g., Diff(55893, 48341)), researchers can see exactly which numbers the model picked and why it chose to subtract them, moving away from "black box" numerical predictions.

Key Results

The results represent a massive leap over the previous state-of-the-art (NAQANet). The model achieved a 33% absolute improvement in Exact Match (EM) score over NAQANet on the DROP dataset, moving from approximately 46.7 to 79.0.

Experimental results on the DROP dataset comparing the proposed model variants against the NAQANet baseline.

The model was particularly effective at "Number" questions (83.0 F1 vs. NAQANet's 45.0) and "Spans" questions, where it correctly identified multiple answers. Furthermore, the authors found that the model could easily adapt to new math word problems. When tested on the Illinois Math Word Problem Dataset, the model reached 83.2% accuracy, outperforming specialized systems by learning to use multiplication and division operators with minimal extra training.

Limitations and Future Work

Despite these gains, the current model is limited to "shallow" programs, typically involving only one or two steps. It may struggle with highly complex, multi-step word problems requiring deep nested logic. Additionally, because the number of possible combinations of numbers and operations is vast, the authors had to prune the search space to the top 128 candidates. Future work may involve more efficient search strategies and more integrated ways of identifying numerical values beyond current heuristic-based extraction.

Summary updated

BibTeX
@inproceedings{andor2019giving,
  author = {Daniel Andor and Luheng He and Kenton Lee and Emily Pitler},
  title = {{Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=VX1ZgbExu93},
}
Figure