FReM: A Flexible Reasoning Mechanism for Balancing Quick and Slow Thinking in Long-Context Question Answering

Zhengyi Zhao, Shubo Zhang, Zezhong Wang, Bin Liang, Binyang Li, Kam-Fai WongarXiv 2025 · arXiv · OpenReview

Abstract

Long-context question-answering (LCQA) systems have greatly benefited from the powerful reasoning capabilities of large language models (LLMs), which can be categorized into slow and quick reasoning modes. However, both modes have their limitations. Slow thinking generally leans to explore every possible reasoning path, which leads to heavy overthinking and wastes time. Quick thinking usually relies on pattern matching rather than truly understanding the query logic, which misses proper understanding. To address these issues, we propose FReM: Flexible Reasoning Mechanism, a method that adjusts reasoning depth according to the complexity of each question. Specifically, FReM leverages synthetic reference QA examples to provide an explicit chain of thought, enabling efficient handling of simple queries while allowing deeper reasoning for more complex ones. By doing so, FReM helps quick-thinking models move beyond superficial pattern matching and narrows the reasoning space for slow-thinking models to avoid unnecessary exploration. Experiments on seven QA datasets show that FReM improves reasoning accuracy and scalability, particularly for complex multihop questions, indicating its potential to advance LCQA methodologies.

Introduction: The Dual-Process Challenge in LCQA

In the field of Large Language Models (LLMs), reasoning is often categorized into two modes inspired by Daniel Kahneman’s dual-process theory: System 1 (Quick) and System 2 (Slow). In Long-Context Question Answering (LCQA), this creates a "Goldilocks" problem.

"Quick" models often rely on superficial pattern matching; while fast, they frequently fail on multi-hop questions because they "underthink" the underlying logic. Conversely, "Slow" models (such as those using Chain-of-Thought) often "overthink," exploring every possible reasoning path even for trivial questions. This leads to "retracing"—where a model realizes it made a wrong turn and must backtrack—wasting computation and increasing the likelihood of errors.

Conceptual comparison of Quick/Slow thinking models versus FReM in multihop and single-step QA scenarios.

Methodology: The FReM Pipeline

FReM (Flexible Reasoning Mechanism) introduces a four-stage pipeline designed to narrow the reasoning space before the model attempts a final answer:

  1. Question Structure Identification: The system decomposes a query into its core structure and "replaceable elements" (placeholders for entities). For example, "Who wrote Pride and Prejudice?" becomes "Who wrote [Book]?".
  2. Synthetic Reference QA Generation: An LLM generates a library of synthetic QA pairs sharing structures with common queries. Crucially, each synthetic example is paired with an explicit, predefined reasoning path (e.g., Deductive, Inductive, or Decompositional).
  3. Selection with Multi-Criteria Matching: When a new question arrives, FReM identifies the most relevant reasoning path from the synthetic library. It uses a scoring formula that balances Skill Coverage (does this path have the right steps?) and Skill Uniqueness (does it use the specific logic this question needs?).
  4. Reasoning Path-Guided Answering: The model is prompted with the original question, the relevant document, and the "best" reasoning path found. This acts as a logical map, guiding the model through the long context.

Experimental Evaluation

The researchers evaluated FReM across seven diverse datasets to test generalizability, including HotpotQA (multi-hop), NewsQA (general domain), and BioASQ (biomedical).

Key Results and Performance

FReM consistently outperformed both quick-thinking (few-shot) and slow-thinking (proactive CoT) baselines. On multihop reasoning tasks, FReM achieved a superior "hits-to-errors" ratio, meaning it identified correct reasoning skills more reliably than models forced to generate long chains of thought from scratch.

Performance comparison showing Hits vs. Errors for various models and methods on multihop reasoning tasks.

Insights into Efficiency

  • Reduced Overthinking: One of the paper's primary contributions is the analysis of "retracing." Slow-thinking models often have retrace rates exceeding 15%. FReM reduces this significantly (down to ~3% in single-step scenarios) by providing a clear plan upfront.
  • Structural Complexity: The authors found that questions with high structural complexity (4+ placeholders) saw the largest performance gains when using FReM compared to standard baselines.
  • Closing the Gap with o1: Using smaller open-source backbones like LLaMA3-8B, FReM-equipped models approached the performance levels of much larger, specialized reasoning models like GPT-o1-mini.

Conclusion

FReM demonstrates that providing models with a "logic map" via synthetic data can effectively balance reasoning depth. By preventing overthinking in simple tasks and providing necessary depth for complex ones, FReM advances the efficiency and accuracy of LCQA systems. Future work may explore the internal attention mechanisms influenced by these guided reasoning chains.

Summary updated

Abstract

Long-context question-answering (LCQA) systems have greatly benefited from the powerful reasoning capabilities of large language models (LLMs), which can be categorized into slow and quick reasoning modes. However, both modes have their limitations. Slow thinking generally leans to explore every possible reasoning path, which leads to heavy overthinking and wastes time. Quick thinking usually relies on pattern matching rather than truly understanding the query logic, which misses proper understanding. To address these issues, we propose FReM: Flexible Reasoning Mechanism, a method that adjusts reasoning depth according to the complexity of each question. Specifically, FReM leverages synthetic reference QA examples to provide an explicit chain of thought, enabling efficient handling of simple queries while allowing deeper reasoning for more complex ones. By doing so, FReM helps quick-thinking models move beyond superficial pattern matching and narrows the reasoning space for slow-thinking models to avoid unnecessary exploration. Experiments on seven QA datasets show that FReM improves reasoning accuracy and scalability, particularly for complex multihop questions, indicating its potential to advance LCQA methodologies.
BibTeX
@misc{zhao2025frem,
  author = {Zhengyi Zhao and Shubo Zhang and Zezhong Wang and Bin Liang and Binyang Li and Kam-Fai Wong},
  title = {{FReM: A Flexible Reasoning Mechanism for Balancing Quick and Slow Thinking in Long-Context Question Answering}},
  year = {2025},
  eprint = {2503.22985},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.22985},
}
Figure