While large language models like BERT have become incredibly good at "extractive" question answering—finding a specific sentence in a text that contains the answer—they often stumble when the answer isn't explicitly written down. If you ask a standard model, "How many touchdowns did the home team score in the second half?" it might find the mentions of touchdowns, but it cannot actually count them.
In their ICLR 2020 paper, researchers from UC Berkeley and Google Brain introduced the Neural Symbolic Reader (NeRd), a framework designed to bridge the gap between neural pattern matching and symbolic logical reasoning.
TL;DR
NeRd is a "domain-agnostic" architecture that combines the linguistic power of BERT with a "programmer" that writes multi-step logical scripts. By treating text spans as variables in a program, NeRd can perform complex operations like counting, sorting, and arithmetic directly on unstructured text, setting new state-of-the-art records on reasoning-heavy datasets like DROP and MathQA.
What Problem Does This Paper Address?
Most Reading Comprehension (RC) models excel at pattern matching but fail at discrete reasoning. To solve a question like "Who kicked the longest field goal?", a model must:
- Identify all field goal yardages and the players who kicked them.
- Associate each player with their specific yardage.
- Compare the numbers to find the maximum.
- Return the name associated with that maximum.
Previous attempts to solve this usually relied on "specialized modules"—one neural network for counting, another for addition, etc. This approach is brittle; it doesn't scale to new types of questions and can't easily handle "compositional" logic (e.g., "Find the average of the two longest field goals"). NeRd aims to replace these specialized "black boxes" with a flexible system that generates and executes transparent programs.
Methods
NeRd replaces specialized modules with a unified architecture that generates executable programs.
NeRd consists of two primary neural components and an execution engine:
- The Reader: A standard transformer (like BERT) that encodes the passage and question into vector representations.
- The Programmer: An LSTM-based decoder with attention. Instead of generating natural language, it generates a sequence of tokens that form a program in a Domain Specific Language (DSL).
- The DSL & Execution: The "secret sauce" is the inclusion of span selection operators (e.g.,
PASSAGE_SPAN).
This allows the symbolic program to "point" back to the text to grab arguments for functions like SUM, COUNT, or ARGMAX.
Training with Weak Supervision
A major hurdle is that most datasets provide the answer (e.g., "3"), but not the program used to get there. The authors use two clever tricks to train NeRd:
- Data Augmentation for Cold Start: They heuristically generate "noisy" programs for simple questions (e.g., turning "Who" into "How many" to create counting examples) to bootstrap the model.
- Thresholded Hard EM: To avoid the "spurious program" problem (where a model finds a mathematically correct but logically nonsensical way to get the right answer), they use an iterative Expectation-Maximization process. They only train on programs that pass a certain probability threshold, which decays as the model becomes more confident.
Datasets
The authors evaluated NeRd on two challenging benchmarks:
- DROP (Discrete Reasoning Over Paragraphs): 96.6K questions requiring math, counting, and sorting based on Wikipedia articles (mostly sports and history).
- MathQA: 37K math word problems that require multiple steps of advanced mathematical reasoning (geometry, physics, probability).
Key Insights and Ideas
- Text as a Database: By introducing operators that select spans, the authors effectively treat unstructured text as a temporary relational database. This avoids the "cascade error" of trying to convert the whole text into a structured table before answering.
- Compositionality is Key: Because the programmer generates a language, it can nest functions. It can calculate
DIFF(VALUE(1), SUM(VALUE(2), VALUE(3)))just as easily as a simple addition. - Domain Agnosticism: Unlike previous models that were hard-coded for the DROP dataset, NeRd's architecture didn't change when moving to MathQA; they simply expanded the library of available symbolic operators (adding things like square roots or constants).
Key Results
- DROP: NeRd achieved a 1.37% absolute gain in Exact Match (EM) and a 1.18% gain in F1 score over the previous state-of-the-art (BERT-Calc).
- MathQA: The results here were dramatic. NeRd outperformed baselines by 25.5% absolute accuracy.
- Data Efficiency: On MathQA, NeRd trained with only 20% of the program annotations still beat the baseline models that had access to 100% of the data.
- Interpretability: Because the model outputs a program, researchers can see why it got an answer wrong (e.g., it picked the wrong span or used a
MINoperator instead of aMAX).
Limitations and Future Work
- DSL Dependency: NeRd still requires humans to define the set of available operators (the DSL) for a given domain. It cannot "invent" a new mathematical concept on its own.
- Search Space: For very long passages with many numbers, the number of possible programs grows exponentially, making the "cold start" problem harder to solve without some initial guidance.
- Execution Errors: The model is only as good as its span selection. If BERT fails to identify the correct number in the text, the symbolic program—no matter how logical—will produce the wrong result.
The authors suggest that future work could involve extending this to even more complex tasks like machine translation or language modeling where symbolic constraints (like grammar rules) could be integrated into the generation process.
Summary updated