Abstract
Defeasible reasoning—the ability to update conclusions based on new evidence—is a hallmark of human intelligence but remains a significant challenge for neural networks. We present CURIOUS, a novel reasoning system that explicitly models question scenarios by generating and refining inference graphs. By employing a hierarchical Mixture of Experts (MoE) to process these graphs, CURIOUS achieves new state-of-the-art results across three major reasoning benchmarks. Our findings demonstrate that "thinking before speaking"—explicitly envisioning a scenario—is as effective for AI as it is for humans.
Introduction
In classical logic, a conclusion derived from a premise is final. However, real-world reasoning is defeasible: conclusions are subject to change as new information arrives. For instance, hearing "The drinking glass fell" typically leads to the conclusion "The glass broke." Yet, if one learns "The glass fell on a pillow," that conclusion is overturned.
While humans naturally form mental models to navigate these shifts, most neural networks reason "reflexively," mapping inputs directly to outputs. This lack of an underlying situational model often leads to failures in complex scenarios. CURIOUS addresses this by explicitly "envisioning" the scenario through a structured inference graph before reaching a final decision.
Methodology
The CURIOUS pipeline consists of three primary stages designed to generate, refine, and encode situational models:
- Graph Generation ($GEN_{init}$): The system uses a sequence-to-sequence approach to generate an initial inference graph. This graph connects the premise, hypothesis, and update through mediating events.
- Graph Correction ($GEN_{corr}$): To handle noise and repetition common in AI-generated structures, a T5-based corrector "cleans" the graph, ensuring semantic clarity and meaningful labels.
- Hierarchical Mixture of Experts (MoE) Encoder: This module serves as the system's core. Unlike standard Graph Convolutional Networks (GCNs), which may propagate noise, CURIOUS utilizes two levels of MoE. MOE-V selectively pools representations of different node types (Contextualizers, Situations, Mediators), while MOE-GX determines the relative weight of the generated graph versus the original text query, allowing the model to ignore unhelpful graphs.
Experiments and Results
CURIOUS was evaluated on three diverse defeasible reasoning datasets: $\delta$-SNLI (natural language inference), $\delta$-SOCIAL (social norms), and $\delta$-ATOMIC (commonsense cause-and-effect). The system established new state-of-the-art (SOTA) performance across all benchmarks:
- $\delta$-SNLI: 85.6% (up from 81.6%)
- $\delta$-SOCIAL: 88.6% (up from 86.2%)
- $\delta$-ATOMIC: 80.2% (up from 78.3%)
The Graph Corrector module was particularly impactful, contributing a ~2-point gain across datasets and emphasizing that the quality of the internal "thought process" is critical.
Discussion and Key Insights
Our analysis revealed several key properties of the CURIOUS system:
- MoE vs. GCN: The MoE approach proved superior for noisy graphs. Its gating mechanism effectively "shuts off" influence from low-quality or irrelevant nodes, whereas standard GCNs tended to propagate errors.
- Asymmetric Reasoning: The model is notably better at using graphs to "strengthen" conclusions than to "weaken" them. This likely reflects a pre-training bias in language models toward predicting likely outcomes rather than negations.
- Emergent Semantic Roles: The MoE learned to group nodes by semantic roles (e.g., background context vs. immediate situation) without explicit supervision.
Conclusion
CURIOUS demonstrates that explicit scenario modeling provides a significant advantage in defeasible reasoning. While the system sets new performance standards, future work will focus on optimizing the interface between the mental model and the final classifier, as well as improving the model's ability to process "weakening" updates.
Summary updated