SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning

Guoxin Chen, Kexin Tang, Chao Yang, Fuying Ye, Yu Qiao, Yiming Qian

Abstract

Elucidating the reasoning process with structured explanations from question to answer is crucial, as it significantly enhances the interpretability, traceability, and trustworthiness of question-answering (QA) systems. However, structured explanations demand models to perform intricately structured reasoning, which poses great challenges. Most existing methods focus on single-step reasoning through supervised learning, ignoring logical dependencies between steps. Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning. In this paper, we propose SEER, a novel method that maximizes a structure-based return to facilitate structured reasoning and explanation. Our proposed structure-based return precisely describes the hierarchical and branching structure inherent in structured reasoning, effectively capturing the intricate relationships between different reasoning steps. In addition, we introduce a fine-grained reward function to meticulously delineate diverse reasoning steps. Extensive experiments show that SEER significantly outperforms state-of-the-art methods, achieving an absolute improvement of 6.9% over RL-based methods on EntailmentBank, a 4.4% average improvement on STREET benchmark, and exhibiting outstanding efficiency and cross-dataset generalization performance.

Introduction: The Challenge of Explainable AI

While Large Language Models (LLMs) have achieved remarkable accuracy in question-answering, they often operate as "black boxes," providing answers without clear justification. To build trustworthy AI, we need systems that provide structured explanations—logical trees or graphs that map out how specific facts lead to a conclusion.

An overview of structured reasoning, showing how a question about natural materials for a table is answered through a hierarchy of facts and intermediate conclusions.

Generating these structures is difficult for current models. Traditional approaches often linearize reasoning into a simple sentence, which ignores the branching nature of logic and leads to "hallucinations." Furthermore, standard reinforcement learning (RL) methods typically assume a "chained" sequence (A → B → C), which fails when multiple independent branches of logic must converge. This misalignment confuses RL agents, as different valid paths to the same logical tree might receive different rewards.

The SEER Framework

To address these issues, we introduce SEER (Structured rEasoning and Explanation via Reinforcement learning). SEER moves beyond linear chains by introducing three core innovations:

1. Structure-based Return

The primary innovation of SEER is the structure-based return. Instead of calculating cumulative rewards based on a linear sequence of actions, SEER evaluates the return based on the actual hierarchy of the reasoning tree or graph. This ensures that the model treats different sequences of steps as equivalent if they result in the same logical structure, leading to more stable and effective training.

A detailed view of the reasoning steps where facts (x) are combined to form intermediate conclusions (i) and finally the hypothesis (h).

2. Fine-grained Reward Function

SEER employs a three-tier reward system to guide the model's exploration:

  • Correct steps (+1): Steps that align perfectly with the ground truth logic.
  • Redundant steps (-0.5): Logically sound steps that do not contribute to the final answer. This encourages exploration without being overly punitive.
  • Erroneous steps (-1): Logically incorrect steps.

3. Generative Policy

Unlike previous RL methods that required exhaustive searching through sentence pairs, SEER uses a generative model (T5-large) to directly produce the next reasoning action. This approach significantly improves efficiency and scalability.

Experimental Results

The researchers evaluated SEER across several rigorous benchmarks, including EntailmentBank (for trees) and the STREET benchmark (for complex graphs).

  • State-of-the-Art Performance: On EntailmentBank, SEER achieved a 6.9% absolute improvement over previous RL-based methods. Notably, it outperformed GPT-4 (using Chain-of-Thought) by 1.9% on the most difficult tasks.
  • Graph Reasoning: On the STREET benchmark, SEER showed a 4.4% average improvement, demonstrating its capability to handle complex, non-linear reasoning structures.
  • Efficiency: By learning a policy to sample actions rather than relying on heavy search algorithms like Monte Carlo Tree Search (MCTS), SEER is approximately 8x faster than previous state-of-the-art methods during inference.
  • Generalization: Cross-dataset tests on eQASC and eOBQA showed that SEER maintains high performance, indicating it learns general logical principles rather than just dataset-specific patterns.

Conclusion and Future Work

SEER represents a significant step toward interpretable and trustworthy QA systems by facilitating structured reasoning through a novel RL framework. While currently limited to text-based inputs, future work aims to extend SEER to multimodal contexts involving images and tables. Despite the multi-step generation process being slower than single-pass models, the gains in transparency and accuracy make SEER a powerful tool for complex reasoning tasks.

Summary updated

Abstract

Elucidating the reasoning process with structured explanations from question to answer is crucial, as it significantly enhances the interpretability, traceability, and trustworthiness of question-answering (QA) systems. However, structured explanations demand models to perform intricately structured reasoning, which poses great challenges. Most existing methods focus on single-step reasoning through supervised learning, ignoring logical dependencies between steps. Moreover, existing reinforcement learning (RL) based methods overlook the structured relationships, underutilizing the potential of RL in structured reasoning. In this paper, we propose SEER, a novel method that maximizes a structure-based return to facilitate structured reasoning and explanation. Our proposed structure-based return precisely describes the hierarchical and branching structure inherent in structured reasoning, effectively capturing the intricate relationships between different reasoning steps. In addition, we introduce a fine-grained reward function to meticulously delineate diverse reasoning steps. Extensive experiments show that SEER significantly outperforms state-of-the-art methods, achieving an absolute improvement of 6.9% over RL-based methods on EntailmentBank, a 4.4% average improvement on STREET benchmark, and exhibiting outstanding efficiency and cross-dataset generalization performance.
BibTeX
@inproceedings{chen2024seer,
  author = {Guoxin Chen and Kexin Tang and Chao Yang and Fuying Ye and Yu Qiao and Yiming Qian},
  title = {{SEER: Facilitating Structured Reasoning and Explanation via Reinforcement Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=TVeC7pnjxO},
}
Figure