Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge

Alon Talmor, Oyvind Tafjord, Peter Clark, Yoav Goldberg, Jonathan Berant

TL;DR

This paper demonstrates that pre-trained language models (LMs) can be taught to systematically combine their internal "implicit" knowledge with new "explicit" facts provided in a prompt. By training on automatically generated reasoning tasks, the authors show that models like RoBERTa can perform complex multi-hop inference and generalize to novel skill combinations, enabling users to correct model errors on-the-fly using simple natural language statements.

What Problem Does This Paper Address?

Large pre-trained language models store a vast "reservoir" of world knowledge within their parameters (implicit knowledge). However, using this knowledge for systematic reasoning—especially when combined with new information—is notoriously difficult to control.

Previous research showed that Transformers could reason over facts explicitly provided in the input (a "closed-world" setup). But real-world applications are "open-domain": a model needs to bridge the gap between what it already knows and what a user tells it. For example, if a model doesn't know that "a whale has a belly button," a user should be able to provide a general rule like "mammals have belly buttons." The model should then "leap" to the correct conclusion by retrieving the implicit fact that "a whale is a mammal."

Figure 1: The model is wrong when asked whether “A whale has a belly button”. However, if a user tells the model the explicit rule “A mammal has a belly button”, the model combines this on-the- fly with its implicit knowledge that “A whale is a mammal”, and arrives at the right conclusion (without re-training).

Methods

The authors propose a procedure to train LMs to perform "Leap-Of-Thought" reasoning. They fine-tune RoBERTa-Large on a multi-task objective using a specific input format: [CLS] explicit knowledge [SEP] hypothesis [SEP].

The core innovation lies in their data generation procedure. To force the model to use implicit knowledge, they create examples where the reasoning chain is incomplete. For a taxonomic reasoning task, they might provide the rule "If X is a mammal, then X has a belly button" but intentionally omit the fact "A whale is a mammal" from the explicit input. The model must retrieve that missing link from its own parameters to verify the hypothesis "A whale has a belly button."

The researchers focused on two high-level reasoning types:

  1. Taxonomic Reasoning: Chaining hypernymy (A is a type of B) or meronymy (A is a part of B) with explicit properties.
  2. Counting: Combining explicit counts with implicit member facts (e.g., counting the members of a band).

Datasets

The authors leveraged existing knowledge bases to automatically generate large-scale training data:

  • ConceptNet, WordNet, and Wikidata: Used to sample (subject, predicate, object) triples for generating facts and rules.
  • Taxonomy Dataset: ~31k examples focused on hypernymy and property inheritance.
  • Counting Dataset: ~39k examples requiring the model to count members of sets (like band members or capitals).
  • RULETAKER & Twenty Questions (20Q): Existing datasets used for multi-task pre-training to establish baseline reasoning and fact-retrieval skills.
  • IMAGINARY Dataset: A controlled benchmark using fake entities (e.g., "A Toucan is a group-1") to ensure the model is actually reasoning rather than relying on memorized answers.
  • MULTI-SKILL-SET: A hand-crafted evaluation set of 185 examples requiring the composition of multiple skills (e.g., age comparison + hypernymy).
Figure 3: Outline of a taxonomy example. The purpose of each relevant and distractor rule is in parenthesis. Underlined hypernym rules are removed in IMPLICIT REASONING.

Key Insights and Ideas

  • Reasoning is Predictable: A model's success in reasoning is highly correlated with its internal "beliefs." If you probe the model and find it knows "A whale is a mammal," it will almost certainly succeed at the reasoning task. If it doesn't know the fact, it fails.
  • One-Shot Correction: Because the reasoning is systematic, users can "fix" a model's mistake by providing the missing fact in the prompt. This works immediately without any further weight updates or retraining.
  • Zero-Shot Skill Transfer: Models trained on hypernymy (IS-A relations) showed a surprising ability to generalize to meronymy (PART-OF relations) without specific training, suggesting they learn a generalized "transitivity" reasoning skill.
  • Compositionality: The most striking finding was that models trained on individual skills (like counting and hypernymy separately) could successfully compose them to solve novel problems that required both skills simultaneously.

Key Results

  • Taxonomic Reasoning: RoBERTa achieved 88.8% accuracy on implicit reasoning tasks, significantly outperforming the "hypothesis-only" baseline of 65.2%.
  • Error Reduction via Intervention: By identifying cases where the model held incorrect implicit beliefs and adding those missing facts to the explicit prompt, the authors reduced the error rate significantly, boosting accuracy from 88.8% to 98.3%.
  • Counting: In tasks where the model had to combine explicit counts with implicit facts, accuracy rose from a baseline of 64.1% to 73%. When all facts were explicit, the model was near-perfect (99.7%).
  • Multi-Skill Generalization: On the complex MULTI-SKILL-SET, the model reached an accuracy of 85.4%, proving it could handle unseen combinations of reasoning types.
Figure 6: An example from the MULTI-SKILL-SET requiring age comparison and hypernymy.

Limitations and Future Work

  • Knowledge Bottleneck: The model is still limited by the breadth and accuracy of its pre-trained implicit knowledge. If a fact isn't in the weights, the "leap" cannot happen.
  • Long-Tail Facts: The model struggled with counting tasks involving obscure, "long-tail" facts that were likely missing from its training corpus.
  • Safety and Trust: The authors note a "Broader Impact" concern: if a model can be taught on-the-fly, a malicious user could provide false rules to manipulate the model's output. Future work is needed to ensure these systems can distinguish between grounded and "fake" information.

Summary updated

Abstract

To what extent can a neural network systematically reason over symbolic facts? Evidence suggests that large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control. Recently, it has been shown that Transformer-based models succeed in consistent reasoning over explicit symbolic facts, under a "closed-world" assumption. However, in an open-domain setup, it is desirable to tap into the vast reservoir of implicit knowledge already encoded in the parameters of pre-trained LMs. In this work, we provide a first demonstration that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements. To do this, we describe a procedure for automatically generating datasets that teach a model new reasoning skills, and demonstrate that models learn to effectively perform inference which involves implicit taxonomic and world knowledge, chaining and counting. Finally, we show that "teaching" models to reason generalizes beyond the training distribution: they successfully compose the usage of multiple reasoning skills in single examples. Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
BibTeX
@inproceedings{talmor2020leap,
  author = {Alon Talmor and Oyvind Tafjord and Peter Clark and Yoav Goldberg and Jonathan Berant},
  title = {{Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=bDZ-XzqoFc5},
}
Figure