SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning

Guoxin Chen, Kexin Tang, Chao Yang, Fuying Ye, Yu Qiao, Yiming Qian

Abstract

Elucidating the reasoning process with structured explanations from question to answer is crucial, as it significantly enhances the interpretability, traceability, and trustworthiness of question-answering (QA) systems. However, structured explanations demand models to perform intricately structured reasoning, which poses great challenges. Most existing methods focus on single-step reasoning through supervised learning, ignoring logical dependencies between steps. Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning. In this paper, we propose SEER, a novel method that maximizes a structure-based return to facilitate structured reasoning and explanation. Our proposed structure-based return precisely describes the hierarchical and branching structure inherent in structured reasoning, effectively capturing the intricate relationships between different reasoning steps. In addition, we introduce a fine-grained reward function to meticulously delineate diverse reasoning steps. Extensive experiments show that SEER significantly outperforms state-of-the-art methods, achieving an absolute improvement of 6.9% over RL-based methods on EntailmentBank, a 4.4% average improvement on STREET benchmark, and exhibiting outstanding efficiency and cross-dataset generalization performance.

Introduction

While Large Language Models (LLMs) are increasingly capable, they often function as "black boxes." Linear "Chain of Thought" (CoT) explanations can be unreliable, occasionally containing hallucinations or logical gaps. To ensure interpretability and trustworthiness, researchers aim for structured reasoning, often represented as entailment trees or reasoning graphs.

An example of structured reasoning where facts are combined into intermediate conclusions to reach a final hypothesis.

The challenge lies in the complexity of generating these structures. Current approaches typically fall into two categories:

  1. Supervised Learning: These models learn to predict individual steps but often ignore long-term logical dependencies.
  2. Standard Reinforcement Learning: Existing RL methods often treat reasoning as a linear chain. However, complex reasoning is rarely a straight line; it is a tree where multiple premises branch out to form intermediate conclusions. Using a "chain-based" reward for a "tree-based" problem leads to training instability.

Methodology: The SEER Approach

SEER (Structured rEasoning and Explanation via Reinforcement learning) introduces three major innovations to bridge this gap:

  • Structure-Based Return: Traditional RL calculates cumulative rewards based on linear sequences. SEER redefines this to follow the actual topology of the reasoning tree. If a conclusion depends on multiple previous steps, the reward for those steps is calculated based on their specific parent-child relationship in the tree, ensuring the model learns the true logical hierarchy.
  • Fine-Grained Reward Function: Most RL models treat any non-optimal step as a failure. SEER distinguishes between correct steps (+1), erroneous steps (-1), and redundant steps (-0.5). By penalizing redundancy less harshly, the model is encouraged to explore the logical space without being paralyzed by the fear of making a non-optimal move.
  • Generative Policy: Instead of calculating probabilities for every possible sentence pair, SEER uses a generative T5 model to directly "write" the next reasoning step. This makes the system significantly faster and more scalable.

Experimental Evaluation

The researchers evaluated SEER across several rigorous benchmarks, including EntailmentBank (focused on entailment trees) and the STREET Benchmark (focused on graph-structured reasoning).

Key Results

  • Outperforming GPT-4: In the most difficult EntailmentBank task (Task 3), SEER achieved an "Overall AllCorrect" score of 12.9%, surpassing GPT-4 (using ReAct) which scored 10.5%.
  • Gains over RL Baselines: SEER showed a 6.9% absolute improvement over previous RL-based methods like RLET on EntailmentBank.
  • Efficiency: SEER’s inference time per tree is 3.91 seconds, compared to 30.77 seconds for the planning-based FAME model—a nearly 8x speedup.

Conclusion

SEER demonstrates that the geometry of the reward system must match the geometry of the logic. By switching from a "chain" reward to a "structure-based" reward, SEER achieves state-of-the-art performance in structured reasoning tasks. Future work will focus on extending this framework to multimodal inputs, integrating information from images and tables into the reasoning process.

Summary updated

Abstract

Elucidating the reasoning process with structured explanations from question to answer is crucial, as it significantly enhances the interpretability, traceability, and trustworthiness of question-answering (QA) systems. However, structured explanations demand models to perform intricately structured reasoning, which poses great challenges. Most existing methods focus on single-step reasoning through supervised learning, ignoring logical dependencies between steps. Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning. In this paper, we propose SEER, a novel method that maximizes a structure-based return to facilitate structured reasoning and explanation. Our proposed structure-based return precisely describes the hierarchical and branching structure inherent in structured reasoning, effectively capturing the intricate relationships between different reasoning steps. In addition, we introduce a fine-grained reward function to meticulously delineate diverse reasoning steps. Extensive experiments show that SEER significantly outperforms state-of-the-art methods, achieving an absolute improvement of 6.9% over RL-based methods on EntailmentBank, a 4.4% average improvement on STREET benchmark, and exhibiting outstanding efficiency and cross-dataset generalization performance. Our code is available at https://github.com/Chen-GX/SEER.
BibTeX
@inproceedings{chen2024seer,
  author = {Guoxin Chen and Kexin Tang and Chao Yang and Fuying Ye and Yu Qiao and Yiming Qian},
  title = {{SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TVeC7pnjxO},
}
Figure