Abstract
Chain-of-Thought (CoT) reasoning enhances the problem-solving ability of large language models (LLMs) but leads to substantial inference overhead, limiting deployment in resource-constrained settings. This paper investigates efficient CoT transfer across models of different scales and architectures through an adaptive reasoning summarization framework. The proposed method compresses reasoning traces via semantic segmentation with importance scoring, budget-aware dynamic compression, and coherence reconstruction, preserving critical reasoning steps while significantly reducing token usage. Experiments on 7,501 medical examination questions across 10 specialties show up to 40% higher accuracy than truncation under the same token budgets. Evaluations on 64 model pairs from eight LLMs (1.5B-32B parameters) confirm strong cross-model transferability. Furthermore, a Gaussian Process-based Bayesian optimization module reduces evaluation cost by 84% and reveals a power-law relationship between model size and cross-domain robustness.
Introduction
The advent of Chain-of-Thought (CoT) prompting has enabled Large Language Models (LLMs) to achieve state-of-the-art performance on complex reasoning tasks by generating intermediate "thinking" steps. However, these reasoning chains are often excessively verbose, frequently exceeding 2,000 tokens for a single query. This verbosity introduces significant latency and cost, making CoT-based inference impractical for edge devices or high-throughput production environments.
While model cascading—using a large "thinking" model to generate CoT and a smaller "answering" model to produce the final output—is a potential solution, naive transfer methods like truncation often break the logical flow, leading to catastrophic performance drops. The core challenge lies in distilling these lengthy reasoning traces into a compact, budget-friendly format without losing the critical insights necessary for correct problem-solving.
The CoT-X Framework
We introduce CoT-X, an adaptive framework designed to optimize cross-model CoT transfer. CoT-X moves beyond simple truncation by employing a sophisticated summarization agent and a Bayesian optimization module for model selection.
The framework operates through a hierarchical compression pipeline:
- Semantic Segmentation & Scoring: Reasoning traces are decomposed into coherent segments. Each segment is assigned an importance score based on its reasoning depth, knowledge density, and relevance to the final conclusion.
- Budget-Aware Selection: A dynamic selection algorithm identifies the most critical logical steps to fit within a predefined token budget (e.g., 64 to 512 tokens).
- Coherence Reconstruction: A summarizer model (e.g., Qwen3-32B) synthesizes the selected segments into a logically bridged, concise reasoning chain.
Additionally, CoT-X utilizes Gaussian Process-based Bayesian Optimization (BO) to efficiently navigate the vast search space of model pairs and compression parameters. This module identifies optimal configurations while reducing the required number of evaluations by 84%.
Experimental Results
The framework was rigorously tested on a medical benchmark comprising 7,501 multiple-choice questions from Japanese National Medical Licensing Examinations, spanning 10 specialties such as Pharmacy, Dentistry, and Internal Medicine. To evaluate cross-lingual performance, the dataset was translated into Chinese and English.
Our evaluation of 64 model pairs (ranging from 1.5B to 32B parameters) yielded several key findings:
- Superior Accuracy under Constraints: At a minimal budget of 64 tokens, CoT-X achieved up to 40% higher accuracy than direct truncation.
- Information Saturation: Diminishing returns were observed for reasoning chains longer than 256 tokens when using smaller answering models, indicating that concise, high-quality logic is more effective than exhaustive traces.
- Robustness Power-Law: We discovered a power-law relationship between model accuracy and cross-domain stability, suggesting that as models improve, their performance across different medical specialties becomes more predictable.
- Efficiency Gains: A 32B "thinker" paired with a 14B "answerer" reached 90% of the peak performance of a full 32B-32B pipeline while utilizing 60% less computation.
Conclusion
CoT-X demonstrates that reasoning summarization is a viable and highly effective path toward efficient CoT transfer. By intelligently compressing reasoning traces, we enable smaller models to inherit the sophisticated logic of their larger counterparts within strict computational budgets. This framework paves the way for advanced, real-time reasoning on edge devices and provides a scalable approach for cloud-edge collaborative AI.
Summary updated