Reasoning Relay: Evaluating Stability and Interchangeability of Large Language Models in Mathematical Reasoning

Leo Lu, Jonathan Zhang, Sean Chua, Spencer Kim, Kevin Zhu, Sean O'Brien, Vasu Sharma

TL;DR

Can one large language model seamlessly pick up a complex reasoning chain where another left off? This paper introduces a "Reasoning Relay" framework to evaluate whether partially completed Chain-of-Thought (CoT) steps can be handed off between different models. The findings reveal a stark architectural divide: while smaller models can successfully finish the thoughts of larger siblings within their own model family, cross-family handoffs frequently fail due to deep-seated representational mismatches.

What Problem Does This Paper Address?

Chain-of-Thought (CoT) prompting has dramatically boosted the mathematical and logical capabilities of LLMs. However, running massive, high-capacity models for long reasoning chains is computationally expensive. A promising alternative is a collaborative, modular framework: use a large, expensive model to tackle the difficult initial setup of a problem, and then hand off the remaining steps to a smaller, cheaper model.

But can LLMs actually pass the reasoning baton mid-stream? Until now, we didn't know if intermediate reasoning traces are truly interchangeable across different model architectures, or if each model family speaks its own distinct "dialect" of logic. This paper investigates this process-level interchangeability to understand inference-time trustworthiness and modular collaboration in AI systems.

Methods

The authors designed a novel chain-splitting framework grounded in token-level probabilities:

  1. Reasoning Generation: A strong base model (e.g., Gemma-3-4B-IT or LLaMA-3.1-70B-Instruct) generates a complete solution to a math problem.
  2. Strategic Truncation: Instead of cutting the text at arbitrary word counts, the authors calculate the cumulative log-probability of the tokens. They truncate the chains at 25%, 50%, and 75% of the total log-probability, capturing early, mid, and late stages of the reasoning process based on the model's internal confidence progression.
  3. Continuation: A smaller continuation model (e.g., Gemma-3-1B-IT or LLaMA-3.1-8B-Instruct) is given the question and the truncated prefix, and tasked with completing the reasoning chain.
  4. PRM Evaluation: To look beyond mere final-answer accuracy, the authors leverage a Process Reward Model (Qwen2.5-PRM) to score the logical coherence and correctness of every single step in the resulting hybrid chain.

They introduce two key metrics: Normalized Relative Gain (NRG), which measures if the handoff helps or hurts performance relative to the standalone model, and Cross-Model Degradation (XMD), which quantifies how severely the continuation model disrupts the original reasoning trajectory.

Datasets

The framework was evaluated exclusively on the test split of the MATH dataset, consisting of 5,000 high-school and college-level mathematical problems spanning diverse topics.

Key Insights and Ideas

  • Models are "Tribal": There is a consistent representational compatibility barrier. A smaller model integrates context smoothly if it comes from a larger model of the same family (e.g., Gemma-4B to Gemma-1B). However, when crossing family lines (e.g., LLaMA-70B to Gemma-1B), the smaller model often fails to maintain correct reasoning, even when handed a highly confident, accurate prefix.
  • Prefixes as Scaffolds: For intra-family handoffs, longer prefixes act as powerful scaffolds. Inheriting a 75% complete chain allows a lightweight model to perform far above its native capacity.
  • Context Integration Overhead: When a smaller model is dropped late into a reasoning chain (at the 75% mark), it faces a "working memory" bottleneck. It must interpret and align with an extensive, externally generated context, which can introduce compounding errors if the stylistic alignment isn't perfect.

Key Results

The baseline performance of each model working entirely on its own can be seen in Table 1:

Baseline performance of individual models on the MATH dataset.

When evaluating the hybrid "relay" chains, the benefits of intra-family alignment become clear. As shown in Table 2, when Gemma-3-4B-IT hands off its reasoning to the tiny Gemma-3-1B-IT, the final accuracy climbs from 41.76% (at 25% truncation) up to 55.26% (at 75% truncation)—a massive leap over the 1B model's standalone accuracy of 36.28%.

Hybrid chain performance using Gemma-3-4B-IT as the base model.

Conversely, Table 3 highlights the friction in cross-family handoffs. When the large LLaMA-3.1-70B-Instruct hands off its reasoning to Gemma-3-1B-IT, the performance degrades significantly. At 25% truncation, the accuracy drops to 36.16% (yielding a negative NRG of -0.1137), showing that the cross-family prefix actually confuses the continuation model.

Hybrid chain performance using LLaMA-3.1-70B-Instruct as the base model.

Limitations and Future Work

  • Single Completion Runs: The experiments relied on deterministic continuations, meaning they did not explore the variance that might occur across multiple stochastic rollouts.
  • Domain Scope: The study was strictly confined to mathematical reasoning. It remains an open question whether these interchangeability patterns hold true for commonsense, scientific, or multimodal tasks.
  • Math-Specific PRMs: The evaluation pipeline utilized a math-specific Process Reward Model; expanding this framework to other domains will require adapting or retraining PRMs for those specific distributions.

Future directions proposed by the authors include exploring adaptive truncation strategies (segmenting chains based on semantic shifts or model uncertainty rather than static thresholds) and deploying these multi-model pipelines into real-world collaborative AI architectures.

Summary updated

Abstract

Chain-of-Thought (CoT) prompting has significantly advanced the reasoning capabilities of large language models (LLMs). While prior work focuses on improving model performance through internal reasoning strategies, little is known about the interchangeability of reasoning across different models. In this work, we explore whether a partially completed reasoning chain from one model can be reliably continued by another model, either within the same model family or across families. We achieve this by assessing the sufficiency of intermediate reasoning traces as transferable scaffolds for logical coherence and final answer accuracy. We interpret this interchangeability as a means of examining inference-time trustworthiness, probing whether reasoning remains both coherent and reliable under model substitution. Using token-level log-probability thresholds to truncate reasoning at early, mid, and late stages from our baseline models, Gemma-3-4B-IT and LLaMA-3.1-70B-Instruct, we conduct continuation experiments with Gemma-3-1B-IT and LLaMA-3.1-8B-Instruct to test intra-family and cross-family behaviors. Our evaluation pipeline leverages truncation thresholds with a Process Reward Model (PRM), providing a reproducible framework for assessing reasoning stability via model interchange. Evaluations with a PRM reveal that hybrid reasoning chains often preserve, and in some cases even improve, final accuracy and logical structure. Our findings point towards interchangeability as an emerging behavioral property of reasoning models, offering insights into new paradigms for reliable modular reasoning in collaborative AI systems.
BibTeX
@misc{lu2025reasoning,
  author = {Leo Lu and Jonathan Zhang and Sean Chua and Spencer Kim and Kevin Zhu and Sean O'Brien and Vasu Sharma},
  title = {{Reasoning Relay: Evaluating Stability and Interchangeability of Large Language Models in Mathematical Reasoning}},
  year = {2025},
  eprint = {2512.20647},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.20647},
}
Figure