SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model

Wencheng Zhang, Shiqin Qiao, Lingjie Luo, Yinfeng Li, Chuanyang Zheng, Qian Xu, Meng Li, Yong Gui, Yijun He, Jianing Qiu, Jindong Hong, Jiankai SunarXiv 2025 · OpenReview

Abstract

With the widespread adoption of large language models (LLMs) in practical applications, selecting an appropriate model requires balancing performance with operational cost. Reasoning-capable models have widened the cost gap between "thinking" (high reasoning) and "non-thinking" (fast, low-cost) modes. We reveal that approximately 58% of medical questions can be accurately answered by the non-thinking mode alone. We propose SynapseRoute, a machine learning-based dynamic routing framework that intelligently assigns input queries to either thinking or non-thinking modes. Experimental results on medical datasets demonstrate that SynapseRoute improves overall accuracy (0.8390 vs. 0.8272) while reducing inference time by 36.8% and token consumption by 39.66%.

Introduction

The recent explosion of reasoning-capable models, such as OpenAI’s o1 or DeepSeek-R1, has introduced a significant trade-off between depth of logic and operational efficiency. While these models excel at complex reasoning, they are often slower and more expensive than standard chat-optimized models. In specialized domains like medicine, query complexity varies significantly—ranging from simple factual recall to complex diagnostic reasoning.

A critical finding of this work is the phenomenon of "over-reasoning." When a model is forced to use a high-reasoning "thinking" mode for a simple query, it may hallucinate unnecessary context or introduce logical noise, leading to incorrect conclusions. SynapseRoute addresses this by dynamically routing queries to the most appropriate inference mode.

The SynapseRoute Framework

SynapseRoute is designed to sit in front of a dual-mode LLM to decide the optimal inference path for each user query.

The SynapseRoute framework compared to single-mode inference, showing significant gains in time, tokens, and accuracy.

The framework consists of three main components:

  1. Automated Data Labeling: The system labels medical questions by comparing the performance of "thinking" and "non-thinking" modes. A "non-thinking" label is assigned if that mode is correct and more efficient, or if the thinking mode fails due to over-reasoning.
  2. The Router: A lightweight classification algorithm (Logistic Regression trained on semantic embeddings) predicts the required mode. This approach achieves a high AUC (0.82) with negligible latency overhead.
  3. AIT Index: To evaluate trade-offs, we introduced the Accuracy-Inference-Token (AIT) Index, a weighted formula ($AIT = a \times A + b \times I + c \times T$) that balances accuracy, latency, and cost according to specific deployment needs.

Experimental Setup and Datasets

The framework was evaluated on 6,365 multiple-choice questions from four authoritative medical sources: USMLE (high reasoning), MedMCQA, PubMedQA, and CareQA (fundamental knowledge). Analysis showed that while USMLE required "thinking" for ~60% of queries, CareQA was dominated by "non-thinking" questions (>80%).

Key Results and Insights

Experimental results demonstrate that SynapseRoute outperforms both static modes by avoiding the pitfalls of over-reasoning while maintaining high efficiency.

Comparison of performance metrics across Non-Thinking, Thinking, and Dynamic (SynapseRoute) modes.
  • Accuracy Gains: SynapseRoute achieved an accuracy of 0.8390, surpassing the Thinking Mode baseline of 0.8272. This 1.5% gain highlights that "thinking more" isn't always better.
  • Efficiency Improvements: Compared to the Thinking Mode, SynapseRoute reduced inference time by 36.8% (10.8s vs 17.1s) and token consumption by 39.66%.
  • The 58% Rule: Approximately 57.8% of medical questions can be answered accurately without high-cost reasoning.
  • AIT Dominance: SynapseRoute consistently achieved the highest AIT scores across various weighting scenarios, proving its robustness to different business constraints.

Conclusion

SynapseRoute demonstrates that intelligent routing can break the traditional trade-off between performance and cost. By identifying queries that do not require deep reasoning, the framework not only saves significant resources but also improves overall accuracy by avoiding the "over-thinking trap." Future work will explore expanding this "cascade" approach to multiple models and broader professional domains.

Summary updated

Abstract

With the widespread adoption of large language models (LLMs) in practical applications, selecting an appropriate model requires balancing not only performance but also operational cost. The emergence of reasoning-capable models has further widened the cost gap between "thinking" (high reasoning) and "non-thinking" (fast, low-cost) modes. In this work, we reveal that approximately 58% of medical questions can be accurately answered by the non-thinking mode alone, without requiring the high-cost reasoning process. This highlights a clear dichotomy in problem complexity and suggests that dynamically routing queries to the appropriate mode based on complexity could optimize accuracy, cost-efficiency, and overall user experience. Based on this, we further propose SynapseRoute, a machine learning-based dynamic routing framework that intelligently assigns input queries to either thinking or non-thinking modes. Experimental results on several medical datasets demonstrate that SynapseRoute not only improves overall accuracy (0.8390 vs. 0.8272) compared to the thinking mode alone but also reduces inference time by 36.8% and token consumption by 39.66%. Importantly, qualitative analysis indicates that over-reasoning on simpler queries can lead to unnecessary delays and even decreased accuracy, a pitfall avoided by our adaptive routing. Finally, this work further introduces the Accuracy-Inference-Token (AIT) index to comprehensively evaluate the trade-offs among accuracy, latency, and token cost.
BibTeX
@misc{zhang2025synapseroute,
  author = {Wencheng Zhang and Shiqin Qiao and Lingjie Luo and Yinfeng Li and Chuanyang Zheng and Qian Xu and Meng Li and Yong Gui and Yijun He and Jianing Qiu and Jindong Hong and Jiankai Sun},
  title = {{SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model}},
  year = {2025},
  eprint = {2507.02822},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.02822},
}
Figure