Beyond Templates: Dynamic Adaptation of Reasoning Demonstrations via Feasibility-Aware Exploration

YongWu, Weihang Pan, Ke Li, Chen Binhui, Ping Li, Binbin LinSubmission to ICLR 2026 · Withdrawn

TL;DR

Directly fine-tuning small language models (SLMs) on long, complex reasoning trajectories generated by expert LLMs often degrades performance because the expert steps exceed the student's capacity. The authors introduce DART (Dynamic Adaptation of Reasoning Trajectories), a framework that detects where an expert trajectory becomes infeasible for an SLM (the "imitation gap") and lets the student autonomously explore an alternative reasoning path that still reaches the correct final answer.

What Problem Does This Paper Address?

Recent work shows that high-quality, step-by-step reasoning demonstrations (like long Chain-of-Thought datasets) can dramatically elicit reasoning capabilities in large models. However, when smaller models (≤3B parameters) are trained via standard supervised fine-tuning (SFT) to mimic these expert trajectories step-by-step, their performance frequently collapses or underperforms zero-shot baselines.

This happens because of a capacity and distributional mismatch: expert trajectories often use abstractions, branching, or verification patterns that the SLM cannot internalize. Blind behavioral cloning forces the student onto brittle, out-of-distribution reasoning paths that it cannot sustain during autoregressive inference.

Methods

DART replaces rigid step-by-step imitation with a three-stage adaptive pipeline:

  1. Step-wise Adaptability Estimation via Solution Simulation: For each step in an expert trajectory, DART estimates an adaptability score $f_t$: the empirical probability that the student model, conditioned on the expert prefix up to step $t$, can successfully generate rollouts that reach the correct ground-truth answer $a^*$.

  2. Imitation Gap Detection: The authors observe that adaptability rises early in the trajectory (as the model benefits from initial setup), but then drops sharply once the expert steps become too complex. When $f_t$ drops significantly below its local maximum ($f_t < f_{t_{\text{peak}}} - \epsilon$), DART halts expert imitation.

  3. Adaptive Path Exploration & Distillation: From the last high-adaptability prefix, the student model autonomously explores alternative reasoning paths using stochastic decoding ($T=0.7$) guided by an outcome-consistency constraint ($O(\tau_{\text{adapt}}) = a^*$). Successful, outcome-consistent trajectories are cleaned (filtering degenerate repetitions) and used for standard cross-entropy fine-tuning.

Expert Trajectory:  [Step 1] -> [Step 2] -> [Step 3] -> [Step 4 (Imitation Gap!)] -> [Answer]
                                              |
Student Exploration:                          +-> [Student Step a] -> [Student Step b] -> [Answer]

Datasets

  • Training / Adaptation Sets:
  • LIMO: 817 high-quality curated mathematical reasoning problems with detailed CoT demonstrations.
  • Math-QwQ-32B: 5,383 MATH training problems with long-form CoT solutions generated by QwQ-32B-Preview.
  • Evaluation Benchmarks:
  • Evaluated across five challenging math benchmarks: GSM8K, GaoKao 2023 En, OlympiadBench, College Math, and MMLU-STEM.

Key Insights and Ideas

  • Empirical and Theoretical Existence of the "Imitation Gap": Adaptability curves consistently display an inverted U-shape. Early expert guidance helps anchor the problem, but deeper expert reasoning introduces steps that fall outside the student's representation space, causing rollout success to plummet.
  • Less Mimicry, Better Reasoning: Enforcing outcome alignment rather than process imitation allows smaller models to discover reasoning paths that match their own latent inductive biases.
  • Lexical Shifts from Hesitation to Directness: Analyzing token distributions showed that expert trajectories frequently contain exploratory and backtracking phrases (e.g., "but", "wait", "alternatively"). Static SFT forces SLMs to mimic this hesitation, confusing them. DART trims this self-doubt, shifting model vocabularies toward decisive, solution-driven terms (e.g., "find", "need", "step").

Key Results

Evaluated on Qwen2.5-Instruct (0.5B, 1.5B, 3B) and LLaMA-3B-Instruct:

  • Static SFT degrades performance: Directly fine-tuning on expert trajectories caused severe performance drops compared to zero-shot (No-Tuning) baselines. For example, Static training on Math-QwQ-32B degraded average accuracy by -8.6% on Qwen2.5-0.5B and -4.6% on Qwen2.5-3B. On LLaMA-3B, static tuning on LIMO dropped performance by -5.5%.
  • DART achieves consistent gains across all scales:
  • On Qwen2.5-3B with LIMO, DART reached an average accuracy of 56.7% (+5.0% over No-Tuning, and +4.5% over Static).
  • On LLaMA-3B with Math-QwQ-32B, DART achieved 35.3% average accuracy (+8.4% over No-Tuning).
  • Ablations validate autonomous search: Intermediate variants like Adaptation-Gap (which merely truncates the expert trajectory at the gap without allowing student exploration) trailed behind Adaptation-Full by 12.1% on 1.5B and 9.8% on 3B models, demonstrating that student exploration is vital to completing the reasoning chain.

Limitations and Future Work

  • Verifiable Tasks Only: The exploration phase strictly requires programmatic or exact-match outcome verification (e.g., math problems with LaTeX boxed answers). The method does not yet extend directly to open-ended, subjective, or unstructured generation tasks where automated outcome verification is challenging.
  • Computational Overhead: Estimating step-wise adaptability requires Monte Carlo rollouts (e.g., 4 rollouts per step across all trajectory steps), which takes around 6 hours on 8× A100 GPUs for a 3B model on the 817-example LIMO dataset.

Summary updated

Abstract

Large language models (LLMs) have shown remarkable reasoning capabilities, yet aligning such abilities to small language models (SLMs) remains a challenge due to distributional mismatches and limited model capacity. Existing reasoning datasets, typically designed for powerful LLMs, often lead to degraded performance when directly applied to weaker models. In this work, we introduce Dynamic Adaptation of Reasoning Trajectories (DART), a novel data adaptation framework that bridges the capability gap between expert reasoning trajectories and diverse SLMs. Instead of uniformly imitating expert steps, DART employs a selective imitation strategy guided by step-wise adaptability estimation via solution simulation. When expert steps surpass the student's capacity—signaled by an Imitation Gap—the student autonomously explores alternative reasoning paths, constrained by outcome consistency. We validate DART across multiple reasoning benchmarks and model scales, demonstrating that it significantly improves generalization and data efficiency over static fine-tuning. Our method enhances supervision quality by aligning training signals with the student’s reasoning capabilities, offering a scalable solution for reasoning alignment in resource-constrained models.
BibTeX
@misc{yongwu2025beyond,
  author = {YongWu and Weihang Pan and Ke Li and Chen Binhui and Ping Li and Binbin Lin},
  title = {{Beyond Templates: Dynamic Adaptation of Reasoning Demonstrations via Feasibility-Aware Exploration}},
  year = {2025},
  eprint = {2505.20700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.20700},
}
Figure