Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension

Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, Quoc V. LeICLR 2020 · OpenReview

While large language models like BERT have become incredibly good at "extractive" question answering—finding a specific sentence in a text that contains the answer—they often stumble when the answer isn't explicitly written down. If you ask a standard model, "How many touchdowns did the home team score in the second half?" it might find the mentions of touchdowns, but it cannot actually count them.

In their ICLR 2020 paper, researchers from UC Berkeley and Google Brain introduced the Neural Symbolic Reader (NeRd), a framework designed to bridge the gap between neural pattern matching and symbolic logical reasoning.

TL;DR

NeRd is a "domain-agnostic" architecture that combines the linguistic power of BERT with a "programmer" that writes multi-step logical scripts. By treating text spans as variables in a program, NeRd can perform complex operations like counting, sorting, and arithmetic directly on unstructured text, setting new state-of-the-art records on reasoning-heavy datasets like DROP and MathQA.

What Problem Does This Paper Address?

Most Reading Comprehension (RC) models excel at pattern matching but fail at discrete reasoning. To solve a question like "Who kicked the longest field goal?", a model must:

  1. Identify all field goal yardages and the players who kicked them.
  2. Associate each player with their specific yardage.
  3. Compare the numbers to find the maximum.
  4. Return the name associated with that maximum.

Previous attempts to solve this usually relied on "specialized modules"—one neural network for counting, another for addition, etc. This approach is brittle; it doesn't scale to new types of questions and can't easily handle "compositional" logic (e.g., "Find the average of the two longest field goals"). NeRd aims to replace these specialized "black boxes" with a flexible system that generates and executes transparent programs.

Methods

NeRd replaces specialized modules with a unified architecture that generates executable programs.

Comparison of NeRd with previous architectures

NeRd consists of two primary neural components and an execution engine:

  1. The Reader: A standard transformer (like BERT) that encodes the passage and question into vector representations.
  2. The Programmer: An LSTM-based decoder with attention. Instead of generating natural language, it generates a sequence of tokens that form a program in a Domain Specific Language (DSL).
  3. The DSL & Execution: The "secret sauce" is the inclusion of span selection operators (e.g., PASSAGE_SPAN).
DSL Operators used in NeRd

This allows the symbolic program to "point" back to the text to grab arguments for functions like SUM, COUNT, or ARGMAX.

Training with Weak Supervision

A major hurdle is that most datasets provide the answer (e.g., "3"), but not the program used to get there. The authors use two clever tricks to train NeRd:

  • Data Augmentation for Cold Start: They heuristically generate "noisy" programs for simple questions (e.g., turning "Who" into "How many" to create counting examples) to bootstrap the model.
  • Thresholded Hard EM: To avoid the "spurious program" problem (where a model finds a mathematically correct but logically nonsensical way to get the right answer), they use an iterative Expectation-Maximization process. They only train on programs that pass a certain probability threshold, which decays as the model becomes more confident.

Datasets

The authors evaluated NeRd on two challenging benchmarks:

  • DROP (Discrete Reasoning Over Paragraphs): 96.6K questions requiring math, counting, and sorting based on Wikipedia articles (mostly sports and history).
  • MathQA: 37K math word problems that require multiple steps of advanced mathematical reasoning (geometry, physics, probability).

Key Insights and Ideas

  • Text as a Database: By introducing operators that select spans, the authors effectively treat unstructured text as a temporary relational database. This avoids the "cascade error" of trying to convert the whole text into a structured table before answering.
  • Compositionality is Key: Because the programmer generates a language, it can nest functions. It can calculate DIFF(VALUE(1), SUM(VALUE(2), VALUE(3))) just as easily as a simple addition.
  • Domain Agnosticism: Unlike previous models that were hard-coded for the DROP dataset, NeRd's architecture didn't change when moving to MathQA; they simply expanded the library of available symbolic operators (adding things like square roots or constants).

Key Results

  • DROP: NeRd achieved a 1.37% absolute gain in Exact Match (EM) and a 1.18% gain in F1 score over the previous state-of-the-art (BERT-Calc).
  • MathQA: The results here were dramatic. NeRd outperformed baselines by 25.5% absolute accuracy.
  • Data Efficiency: On MathQA, NeRd trained with only 20% of the program annotations still beat the baseline models that had access to 100% of the data.
  • Interpretability: Because the model outputs a program, researchers can see why it got an answer wrong (e.g., it picked the wrong span or used a MIN operator instead of a MAX).

Limitations and Future Work

  • DSL Dependency: NeRd still requires humans to define the set of available operators (the DSL) for a given domain. It cannot "invent" a new mathematical concept on its own.
  • Search Space: For very long passages with many numbers, the number of possible programs grows exponentially, making the "cold start" problem harder to solve without some initial guidance.
  • Execution Errors: The model is only as good as its span selection. If BERT fails to identify the correct number in the text, the symbolic program—no matter how logical—will produce the wrong result.

The authors suggest that future work could involve extending this to even more complex tasks like machine translation or language modeling where symbolic constraints (like grammar rules) could be integrated into the generation process.

Summary updated

Abstract

Integrating distributed representations with symbolic operations is essential for reading comprehension requiring complex reasoning, such as counting, sorting and arithmetics, but most existing approaches are hard to scale to more domains or more complex reasoning. In this work, we propose the Neural Symbolic Reader (NeRd), which includes a reader, e.g., BERT, to encode the passage and question, and a programmer, e.g., LSTM, to generate a program that is executed to produce the answer. Compared to previous works, NeRd is more scalable in two aspects: (1) domain-agnostic, i.e., the same neural architecture works for different domains; (2) compositional, i.e., when needed, complex programs can be generated by recursively applying the predefined operators, which become executable and interpretable representations for more complex reasoning. Furthermore, to overcome the challenge of training NeRd with weak supervision, we apply data augmentation techniques and hard Expectation-Maximization (EM) with thresholding. On DROP, a challenging reading comprehension dataset that requires discrete reasoning, NeRd achieves 1.37%/1.18% absolute improvement over the state-of-the-art on EM/F1 metrics. With the same architecture, NeRd significantly outperforms the baselines on MathQA, a math problem benchmark that requires multiple steps of reasoning, by 25.5% absolute increment on accuracy when trained on all the annotated programs. More importantly, NeRd still beats the baselines even when only 20% of the program annotations are given.
BibTeX
@inproceedings{chen2020neural,
  author = {Xinyun Chen and Chen Liang and Adams Wei Yu and Denny Zhou and Dawn Song and Quoc V. Le},
  title = {{Neural Symbolic Reader: Scalable Integration of Distributed and Symbolic Representations for Reading Comprehension}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=efBWC4rQZX2},
}
Figure