TL;DR
Can one large language model seamlessly pick up a complex reasoning chain where another left off? This paper introduces a "Reasoning Relay" framework to evaluate whether partially completed Chain-of-Thought (CoT) steps can be handed off between different models. The findings reveal a stark architectural divide: while smaller models can successfully finish the thoughts of larger siblings within their own model family, cross-family handoffs frequently fail due to deep-seated representational mismatches.
What Problem Does This Paper Address?
Chain-of-Thought (CoT) prompting has dramatically boosted the mathematical and logical capabilities of LLMs. However, running massive, high-capacity models for long reasoning chains is computationally expensive. A promising alternative is a collaborative, modular framework: use a large, expensive model to tackle the difficult initial setup of a problem, and then hand off the remaining steps to a smaller, cheaper model.
But can LLMs actually pass the reasoning baton mid-stream? Until now, we didn't know if intermediate reasoning traces are truly interchangeable across different model architectures, or if each model family speaks its own distinct "dialect" of logic. This paper investigates this process-level interchangeability to understand inference-time trustworthiness and modular collaboration in AI systems.
Methods
The authors designed a novel chain-splitting framework grounded in token-level probabilities:
- Reasoning Generation: A strong base model (e.g., Gemma-3-4B-IT or LLaMA-3.1-70B-Instruct) generates a complete solution to a math problem.
- Strategic Truncation: Instead of cutting the text at arbitrary word counts, the authors calculate the cumulative log-probability of the tokens. They truncate the chains at 25%, 50%, and 75% of the total log-probability, capturing early, mid, and late stages of the reasoning process based on the model's internal confidence progression.
- Continuation: A smaller continuation model (e.g., Gemma-3-1B-IT or LLaMA-3.1-8B-Instruct) is given the question and the truncated prefix, and tasked with completing the reasoning chain.
- PRM Evaluation: To look beyond mere final-answer accuracy, the authors leverage a Process Reward Model (Qwen2.5-PRM) to score the logical coherence and correctness of every single step in the resulting hybrid chain.
They introduce two key metrics: Normalized Relative Gain (NRG), which measures if the handoff helps or hurts performance relative to the standalone model, and Cross-Model Degradation (XMD), which quantifies how severely the continuation model disrupts the original reasoning trajectory.
Datasets
The framework was evaluated exclusively on the test split of the MATH dataset, consisting of 5,000 high-school and college-level mathematical problems spanning diverse topics.
Key Insights and Ideas
- Models are "Tribal": There is a consistent representational compatibility barrier. A smaller model integrates context smoothly if it comes from a larger model of the same family (e.g., Gemma-4B to Gemma-1B). However, when crossing family lines (e.g., LLaMA-70B to Gemma-1B), the smaller model often fails to maintain correct reasoning, even when handed a highly confident, accurate prefix.
- Prefixes as Scaffolds: For intra-family handoffs, longer prefixes act as powerful scaffolds. Inheriting a 75% complete chain allows a lightweight model to perform far above its native capacity.
- Context Integration Overhead: When a smaller model is dropped late into a reasoning chain (at the 75% mark), it faces a "working memory" bottleneck. It must interpret and align with an extensive, externally generated context, which can introduce compounding errors if the stylistic alignment isn't perfect.
Key Results
The baseline performance of each model working entirely on its own can be seen in Table 1:
When evaluating the hybrid "relay" chains, the benefits of intra-family alignment become clear. As shown in Table 2, when Gemma-3-4B-IT hands off its reasoning to the tiny Gemma-3-1B-IT, the final accuracy climbs from 41.76% (at 25% truncation) up to 55.26% (at 75% truncation)—a massive leap over the 1B model's standalone accuracy of 36.28%.
Conversely, Table 3 highlights the friction in cross-family handoffs. When the large LLaMA-3.1-70B-Instruct hands off its reasoning to Gemma-3-1B-IT, the performance degrades significantly. At 25% truncation, the accuracy drops to 36.16% (yielding a negative NRG of -0.1137), showing that the cross-family prefix actually confuses the continuation model.
Limitations and Future Work
- Single Completion Runs: The experiments relied on deterministic continuations, meaning they did not explore the variance that might occur across multiple stochastic rollouts.
- Domain Scope: The study was strictly confined to mathematical reasoning. It remains an open question whether these interchangeability patterns hold true for commonsense, scientific, or multimodal tasks.
- Math-Specific PRMs: The evaluation pipeline utilized a math-specific Process Reward Model; expanding this framework to other domains will require adapting or retraining PRMs for those specific distributions.
Future directions proposed by the authors include exploring adaptive truncation strategies (segmenting chains based on semantic shifts or model uncertainty rather than static thresholds) and deploying these multi-model pipelines into real-world collaborative AI architectures.
Summary updated