T2: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering.

Zhengyi Zhao, Shubo Zhang, Zezhong Wang, Huimin Wang, Yutian Zhao, Bin Liang, Yefeng Zheng, Binyang Li, Kam-Fai Wong, Xian Wu

Abstract

Contextual Question Answering (CQA) often suffers from an imbalance between reasoning depth and computational efficiency. We present Think-to-Think ($T^2$), a novel framework that enables Large Language Models (LLMs) to dynamically scale their reasoning based on question complexity. By generating synthetic examples to determine optimal reasoning strategies, $T^2$ improves accuracy while reducing computational overhead by up to 25.2% and inference time by 56.3% compared to existing slow-thinking methods.

Introduction

Current CQA systems face a significant "accuracy-vs-efficiency" trade-off. Direct answering often fails on complex, multi-hop questions, while uniform Chain-of-Thought (CoT) reasoning leads to "overthinking" on simple tasks. This overthinking results in verbose, redundant outputs that are not only costly but can also degrade performance. Existing Test-Time Scaling (TTS) methods typically rely on fixed budgets or early-stopping, which often fail to capture the inherent difficulty of a task. $T^2$ addresses this by allowing the model to self-evaluate and adapt its reasoning strategy at test time.

Methodology

The $T^2$ framework operates through a four-stage pipeline designed to match the reasoning depth to the specific requirements of the input question.

Comparison of reasoning frameworks: (a) Direct Prompt & CoT, (b) Adaptive-of-Thought, and (c) the proposed T2 framework.
  1. Question Decomposition: The system uses a fine-tuned RoBERTa model to extract the "skeleton" of a question, separating structural tokens from specific entities.
  2. Similar Examples Generation: The LLM generates synthetic questions based on the extracted structure and constructs corresponding reasoning strategies (sub-questions and specific reasoning skills).
  3. Multi-Criteria Matching: The system evaluates synthetic examples using Skill Coverage and Skill Uniqueness to identify the most appropriate strategy for the original question.
  4. Strategy-Guided Answering: The selected strategy guides the model to extract relevant document segments and produce a concise answer.

Experimental Evaluation

The framework was validated across seven diverse CQA benchmarks, including SQuAD, HotpotQA, BioASQ, NewsQA, GAOKAO, HQA, and TriviaQA. These datasets cover general knowledge, multi-hop reasoning, specialized domains (biomedical, history), and academic testing.

Results and Discussion

$T^2$ consistently outperforms traditional baselines in both accuracy and efficiency. For example, on the SQuAD dataset, $T^2$ improved the ROUGE-L score of Qwen2.5-32B from 73.41 to 81.86.

Hits and Errors of Complex QA, demonstrating that T2 achieves superior hit rates while maintaining low error rates compared to other methods.

Key findings include:

  • Efficiency Gains: $T^2$ reduced token consumption by up to 25.2% compared to QwQ-32B-Preview and accelerated inference by 56.3% over MCTS-based methods.
  • Reasoning Quality: The framework significantly reduces "retracing" (mid-output revisions), leading to cleaner reasoning paths. As shown in the performance analysis, $T^2$ achieves a higher "Hit" rate in retrieving required facts while minimizing the introduction of spurious information (Error rate).
  • Adaptability: The "Think-to-Think" principle allows the model to recognize when a direct approach is sufficient, avoiding the pitfalls of forced long-chain reasoning.

Conclusion and Future Work

$T^2$ demonstrates that adaptive test-time scaling is a viable path toward more efficient and accurate LLM reasoning. While highly effective in textual CQA, future work will explore extending this framework to multimodal contexts and addressing potential structural biases in synthetic example generation.

Summary updated

Abstract

Recent advances in Large Language Models (LLMs) have demonstrated remarkable performance in Contextual Question Answering (CQA). However, prior approaches typically employ elaborate reasoning strategies regardless of question complexity, leading to low adaptability. Recent efficient test-time scaling methods introduce budget constraints or early stop mechanisms to avoid overthinking for straightforward questions. But they add human bias to the reasoning process and fail to leverage models' inherent reasoning capabilities. To address these limitations, we present T$^2$: Think-to-Think, a novel framework that dynamically adapts reasoning depth based on question complexity. T$^2$ leverages the insight that if an LLM can effectively solve similar questions using specific reasoning strategies, it can apply the same strategy to the original question. This insight enables to adoption of concise reasoning for straightforward questions while maintaining detailed analysis for complex problems. T$^2$ works through four key steps: decomposing questions into structural elements, generating similar examples with candidate reasoning strategies, evaluating these strategies against multiple criteria, and applying the most appropriate strategy to the original question. Experimental evaluation across seven diverse CQA benchmarks demonstrates that T$^2$ not only achieves higher accuracy than baseline methods but also reduces computational overhead by up to 25.2\%.
BibTeX
@inproceedings{zhao2025t,
  author = {Zhengyi Zhao and Shubo Zhang and Zezhong Wang and Huimin Wang and Yutian Zhao and Bin Liang and Yefeng Zheng and Binyang Li and Kam-Fai Wong and Xian Wu},
  title = {{T2: An Adaptive Test-Time Scaling Strategy for Contextual Question Answering.}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=u2bBYe0rgQ},
}
Figure