Think about it! Improving defeasible reasoning by first modeling the question scenario

Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, Eduard H. Hovy

Abstract

Defeasible reasoning—the ability to update conclusions based on new evidence—is a hallmark of human intelligence but remains a significant challenge for neural networks. We present CURIOUS, a novel reasoning system that explicitly models question scenarios by generating and refining inference graphs. By employing a hierarchical Mixture of Experts (MoE) to process these graphs, CURIOUS achieves new state-of-the-art results across three major reasoning benchmarks. Our findings demonstrate that "thinking before speaking"—explicitly envisioning a scenario—is as effective for AI as it is for humans.

Introduction

In classical logic, a conclusion derived from a premise is final. However, real-world reasoning is defeasible: conclusions are subject to change as new information arrives. For instance, hearing "The drinking glass fell" typically leads to the conclusion "The glass broke." Yet, if one learns "The glass fell on a pillow," that conclusion is overturned.

Figure 1: An inference graph illustrating how new evidence (falling on a pillow) can weaken the hypothesis that a glass broke.

While humans naturally form mental models to navigate these shifts, most neural networks reason "reflexively," mapping inputs directly to outputs. This lack of an underlying situational model often leads to failures in complex scenarios. CURIOUS addresses this by explicitly "envisioning" the scenario through a structured inference graph before reaching a final decision.

Methodology

The CURIOUS pipeline consists of three primary stages designed to generate, refine, and encode situational models:

  1. Graph Generation ($GEN_{init}$): The system uses a sequence-to-sequence approach to generate an initial inference graph. This graph connects the premise, hypothesis, and update through mediating events.
  2. Graph Correction ($GEN_{corr}$): To handle noise and repetition common in AI-generated structures, a T5-based corrector "cleans" the graph, ensuring semantic clarity and meaningful labels.
  3. Hierarchical Mixture of Experts (MoE) Encoder: This module serves as the system's core. Unlike standard Graph Convolutional Networks (GCNs), which may propagate noise, CURIOUS utilizes two levels of MoE. MOE-V selectively pools representations of different node types (Contextualizers, Situations, Mediators), while MOE-GX determines the relative weight of the generated graph versus the original text query, allowing the model to ignore unhelpful graphs.
Figure 2: The CURIOUS architecture, showing the progression from text input to a corrected inference graph and final classification.

Experiments and Results

CURIOUS was evaluated on three diverse defeasible reasoning datasets: $\delta$-SNLI (natural language inference), $\delta$-SOCIAL (social norms), and $\delta$-ATOMIC (commonsense cause-and-effect). The system established new state-of-the-art (SOTA) performance across all benchmarks:

  • $\delta$-SNLI: 85.6% (up from 81.6%)
  • $\delta$-SOCIAL: 88.6% (up from 86.2%)
  • $\delta$-ATOMIC: 80.2% (up from 78.3%)

The Graph Corrector module was particularly impactful, contributing a ~2-point gain across datasets and emphasizing that the quality of the internal "thought process" is critical.

Discussion and Key Insights

Our analysis revealed several key properties of the CURIOUS system:

  • MoE vs. GCN: The MoE approach proved superior for noisy graphs. Its gating mechanism effectively "shuts off" influence from low-quality or irrelevant nodes, whereas standard GCNs tended to propagate errors.
  • Asymmetric Reasoning: The model is notably better at using graphs to "strengthen" conclusions than to "weaken" them. This likely reflects a pre-training bias in language models toward predicting likely outcomes rather than negations.
  • Emergent Semantic Roles: The MoE learned to group nodes by semantic roles (e.g., background context vs. immediate situation) without explicit supervision.

Conclusion

CURIOUS demonstrates that explicit scenario modeling provides a significant advantage in defeasible reasoning. While the system sets new performance standards, future work will focus on optimizing the interface between the mental model and the final classifier, as well as improving the model's ability to process "weakening" updates.

Summary updated

Abstract

Defeasible reasoning is the mode of reasoning where conclusions can be overturned by taking into account new evidence. Existing cognitive science literature on defeasible reasoning suggests that a person forms a mental model of the problem scenario before answering questions. Our research goal asks whether neural models can similarly benefit from envisioning the question scenario before answering a defeasible query. Our approach is, given a question, to have a model first create a graph of relevant influences, and then leverage that graph as an additional input when answering the question. Our system, CURIOUS, achieves a new state-of-the-art on three different defeasible reasoning datasets. This result is significant as it illustrates that performance can be improved by guiding a system to "think about" a question and explicitly model the scenario, rather than answering reflexively. Code, data, and pre-trained models are located at https://github.com/madaan/thinkaboutit.
BibTeX
@inproceedings{madaan2021think,
  author = {Aman Madaan and Niket Tandon and Dheeraj Rajagopal and Peter Clark and Yiming Yang and Eduard H. Hovy},
  title = {{Think about it! Improving defeasible reasoning by first modeling the question scenario}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=C8JHYuvleLm},
}
Figure