Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers

Wanjun Zhong, Tingting Ma, Jiahai Wang, Jian Yin, Tiejun Zhao, Chin-Yew Lin, Nan Duan

Abstract

This paper presents ReasonFormer, a unified reasoning framework designed to mirror the modular and compositional reasoning process of humans in complex decision-making. Inspired by dual-process theory in cognitive science, ReasonFormer decouples the representation module (System 1: automatic thinking) from reasoning modules (System 2: controlled thinking) to capture different levels of cognition. Built upon the representation module, specialized reasoning modules are pre-trained in fundamental skills such as logic and extraction. To mimic controlled compositional thinking, these modules are dynamically activated and composed in both parallel and cascaded manners. Evaluated on 11 datasets, ReasonFormer demonstrates substantial performance boosts and superior generalization in few-shot learning, revealing the power of modular, compositional reasoning.

Introduction: The Need for Modular Reasoning

Prevailing Large Language Models (LLMs) typically employ a "homogeneous" architecture, processing every input through a fixed, monolithic sequence of layers. While effective for many tasks, this approach struggles to explicitly model the complex, modular reasoning processes humans use.

In cognitive science, Dual-Process Theory posits that human cognition consists of two systems: System 1 (fast, intuitive, and automatic) and System 2 (slow, logical, and controlled). Current LLMs essentially collapse these two systems into a single computational "call." This makes it difficult for models to handle tasks requiring specific, cascaded skills—such as recalling a factual detail, performing a logical deduction based on that detail, and then extracting a final answer. The authors of ReasonFormer aim to "disentangle" these capabilities, creating a model that can dynamically decide which skills to use and how "deeply" to process any given problem.

The ReasonFormer Framework

ReasonFormer is a unified framework built on an encoder-decoder (T5) architecture, introducing several novel components to achieve modularity and compositionality.

The ReasonFormer architecture featuring decoupled representation and reasoning modules.

Decoupled Modules

The encoder is split into two distinct parts. The first nine layers function as the Representation Module (System 1), focusing on contextual understanding and semantic representation. The final layers are replaced by a set of specialized Reasoning Modules (RMs) (System 2).

Specialized Reasoning Modules

The RMs are pre-trained to be "professionals" in five fundamental reasoning skills:

  • Logic: For deductive reasoning.
  • Simple QA: For answer extraction.
  • NER: For named entity recognition.
  • NLI: For determining semantic relevance and natural language inference.
  • Factual Knowledge: For memorization and retrieval of facts.

Dynamic Composition and Control

For every input, a Skill Router dynamically calculates which RMs to activate at each step. Simultaneously, a Stop Gate decides whether the current reasoning is sufficient or if another "cascaded" step is required. Unlike standard Mixture-of-Experts (MoE) models that activate experts in parallel, ReasonFormer can stack these modules. For example, it might pass information through a "Fact" module in step one and a "Logic" module in step two, allowing for complex, multi-step reasoning chains.

Experimental Setup

The model was evaluated on 11 diverse datasets requiring various reasoning skills:

  • Logical Reasoning: ReClor
  • Commonsense Knowledge: CSQA, ARC, PIQA, HellaSwag
  • Natural Language Inference: aNLI
  • Multi-hop QA: HotpotQA, WikiHop
  • Dialogue Reasoning: MuTual, DREAM
  • General Reading Comprehension: RACE

The specialized RMs were pre-trained using a mix of self-supervised and semi-supervised corpora, including LogicGAN for logic, ProQA for QA, and Wikidata for facts and NER.

Key Insights

Modularity and Generalization

Few-shot experiments revealed that specialized modules could be "frozen" while still allowing the model to adapt to new tasks. By keeping the reasoning modules fixed and only training the router and adapters, the model demonstrated that it had truly learned "skills" that can be repurposed without catastrophic forgetting.

The Importance of Depth

Analysis of the "Stop Gate" showed that different tasks require varying reasoning depths. Simpler tasks often stop early, while complex multi-hop questions consistently utilize the maximum reasoning depth, mirroring human deliberate thinking.

Cascaded vs. Parallel Reasoning

ReasonFormer significantly outperformed a Mixture-of-Reasoning-Modules (MoRM) baseline that only uses parallel experts. This suggests that the ability to sequence different skills is more critical for complex reasoning than simply having a variety of skills available simultaneously.

Key Results

ReasonFormer demonstrated state-of-the-art performance, outperforming both the vanilla T5 base and a "Reasoning Pre-Trained" T5 (RPT-T5) across all 11 datasets.

Performance comparison of ReasonFormer against T5 and MoRM baselines across 11 datasets.

On average, ReasonFormer achieved a score of 62.1, a significant improvement over the 54.3 of Vanilla T5 and 57.2 of RPT-T5. Furthermore, the model maintained high efficiency; using "Sparse Activation" (top-2 modules per step), ReasonFormer achieved these results while reducing the number of activated parameters compared to full activation.

Conclusion and Future Directions

ReasonFormer validates the cognitive science-inspired approach of decoupling intuitive representation from deliberate reasoning. By modularizing reasoning skills, the framework achieves better performance, efficiency, and few-shot generalization.

Future work may involve expanding the library of reasoning modules to include specialized domains like numerical reasoning or code. Additionally, exploring how this modularity performs in decoder-only architectures (like GPT) could further enhance the scalability and applicability of compositional reasoning in large-scale language models.

Summary updated

Abstract

This paper presents ReasonFormer, a unified reasoning framework for mirroring the modular and compositional reasoning process of humans in complex decision-making. Inspired by dual-process theory in cognitive science, the representation module (automatic thinking) and reasoning modules (controlled thinking) are decoupled to capture different levels of cognition. Upon the top of the representation module, the pre-trained reasoning modules are modular and professional in specific and fundamental reasoning skills (e.g., logic, simple QA, etc). To mimic the controlled compositional thinking process, different reasoning modules are dynamically activated and composed in both parallel and cascaded manners to control what reasoning skills are activated and how deep the reasoning process will be reached to solve the current problems. The unified reasoning framework solves multiple tasks with a single model, and is trained and inferred in an end-to-end manner. Evaluated on 11 datasets requiring different reasoning skills and complexity, ReasonFormer demonstrates substantial performance boosts, revealing the compositional reasoning ability. Few-shot experiments exhibit better generalization ability by learning to compose pre-trained skills for new tasks with limited data, and decoupling the representation module and the reasoning modules. Further analysis shows the modularity of reasoning modules as different tasks activate distinct reasoning skills at different reasoning depths.
BibTeX
@inproceedings{zhong2023disentangling,
  author = {Wanjun Zhong and Tingting Ma and Jiahai Wang and Jian Yin and Tiejun Zhao and Chin-Yew Lin and Nan Duan},
  title = {{Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=rdJQJ9ZQhq7},
}
Figure