TL;DR
This paper demonstrates that pre-trained language models (LMs) can be taught to systematically combine their internal "implicit" knowledge with new "explicit" facts provided in a prompt. By training on automatically generated reasoning tasks, the authors show that models like RoBERTa can perform complex multi-hop inference and generalize to novel skill combinations, enabling users to correct model errors on-the-fly using simple natural language statements.
What Problem Does This Paper Address?
Large pre-trained language models store a vast "reservoir" of world knowledge within their parameters (implicit knowledge). However, using this knowledge for systematic reasoning—especially when combined with new information—is notoriously difficult to control.
Previous research showed that Transformers could reason over facts explicitly provided in the input (a "closed-world" setup). But real-world applications are "open-domain": a model needs to bridge the gap between what it already knows and what a user tells it. For example, if a model doesn't know that "a whale has a belly button," a user should be able to provide a general rule like "mammals have belly buttons." The model should then "leap" to the correct conclusion by retrieving the implicit fact that "a whale is a mammal."
Methods
The authors propose a procedure to train LMs to perform "Leap-Of-Thought" reasoning. They fine-tune RoBERTa-Large on a multi-task objective using a specific input format: [CLS] explicit knowledge [SEP] hypothesis [SEP].
The core innovation lies in their data generation procedure. To force the model to use implicit knowledge, they create examples where the reasoning chain is incomplete. For a taxonomic reasoning task, they might provide the rule "If X is a mammal, then X has a belly button" but intentionally omit the fact "A whale is a mammal" from the explicit input. The model must retrieve that missing link from its own parameters to verify the hypothesis "A whale has a belly button."
The researchers focused on two high-level reasoning types:
- Taxonomic Reasoning: Chaining hypernymy (A is a type of B) or meronymy (A is a part of B) with explicit properties.
- Counting: Combining explicit counts with implicit member facts (e.g., counting the members of a band).
Datasets
The authors leveraged existing knowledge bases to automatically generate large-scale training data:
- ConceptNet, WordNet, and Wikidata: Used to sample (subject, predicate, object) triples for generating facts and rules.
- Taxonomy Dataset: ~31k examples focused on hypernymy and property inheritance.
- Counting Dataset: ~39k examples requiring the model to count members of sets (like band members or capitals).
- RULETAKER & Twenty Questions (20Q): Existing datasets used for multi-task pre-training to establish baseline reasoning and fact-retrieval skills.
- IMAGINARY Dataset: A controlled benchmark using fake entities (e.g., "A Toucan is a group-1") to ensure the model is actually reasoning rather than relying on memorized answers.
- MULTI-SKILL-SET: A hand-crafted evaluation set of 185 examples requiring the composition of multiple skills (e.g., age comparison + hypernymy).
Key Insights and Ideas
- Reasoning is Predictable: A model's success in reasoning is highly correlated with its internal "beliefs." If you probe the model and find it knows "A whale is a mammal," it will almost certainly succeed at the reasoning task. If it doesn't know the fact, it fails.
- One-Shot Correction: Because the reasoning is systematic, users can "fix" a model's mistake by providing the missing fact in the prompt. This works immediately without any further weight updates or retraining.
- Zero-Shot Skill Transfer: Models trained on hypernymy (IS-A relations) showed a surprising ability to generalize to meronymy (PART-OF relations) without specific training, suggesting they learn a generalized "transitivity" reasoning skill.
- Compositionality: The most striking finding was that models trained on individual skills (like counting and hypernymy separately) could successfully compose them to solve novel problems that required both skills simultaneously.
Key Results
- Taxonomic Reasoning: RoBERTa achieved 88.8% accuracy on implicit reasoning tasks, significantly outperforming the "hypothesis-only" baseline of 65.2%.
- Error Reduction via Intervention: By identifying cases where the model held incorrect implicit beliefs and adding those missing facts to the explicit prompt, the authors reduced the error rate significantly, boosting accuracy from 88.8% to 98.3%.
- Counting: In tasks where the model had to combine explicit counts with implicit facts, accuracy rose from a baseline of 64.1% to 73%. When all facts were explicit, the model was near-perfect (99.7%).
- Multi-Skill Generalization: On the complex MULTI-SKILL-SET, the model reached an accuracy of 85.4%, proving it could handle unseen combinations of reasoning types.
Limitations and Future Work
- Knowledge Bottleneck: The model is still limited by the breadth and accuracy of its pre-trained implicit knowledge. If a fact isn't in the weights, the "leap" cannot happen.
- Long-Tail Facts: The model struggled with counting tasks involving obscure, "long-tail" facts that were likely missing from its training corpus.
- Safety and Trust: The authors note a "Broader Impact" concern: if a model can be taught on-the-fly, a malicious user could provide false rules to manipulate the model's output. Future work is needed to ensure these systems can distinguish between grounded and "fake" information.
Summary updated