Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework

Jie Chen, Jianping Jiang, Yingqian Min, Zican Dong, Shijie Wang, Xin Zhao, Ji-Rong Wen2025 · DOI

Abstract

Large reasoning models (LRMs) have exhibited strong performance on complex reasoning tasks, with further gains achievable through increased computational budgets at inference. However, current test-time scaling methods predominantly rely on redundant sampling, ignoring the historical experience utilization, thereby limiting computational efficiency. To overcome this limitation, we propose Sticker-TTS, a novel test-time scaling framework that coordinates three collaborative LRMs to iteratively explore and refine solutions guided by historical attempts. At the core of our framework are distilled key conditions—termed stickers—which drive the extraction, refinement, and reuse of critical information across multiple rounds of reasoning. To further enhance the efficiency and performance of our framework, we introduce a two-stage optimization strategy that combines imitation learning with self-improvement, enabling progressive refinement. Extensive evaluations on three challenging mathematical reasoning benchmarks, including AIME-24, AIME-25, and Olym-MATH, demonstrate that Sticker-TTS consistently surpasses strong baselines, including self-consistency and advanced reinforcement learning approaches, under comparable inference budgets.

Introduction

Large Reasoning Models (LRMs) have demonstrated that increasing computational resources during inference—a process known as test-time scaling—can significantly improve performance on complex tasks. However, traditional scaling methods like "Self-Consistency" (Cons@N) often treat each reasoning attempt as an isolated event, leading to computational redundancy and a failure to learn from previous mistakes. Conversely, iterative methods that feed back entire reasoning chains often suffer from information overload, where the model becomes lost in its own verbose history.

Sticker-TTS addresses these challenges by introducing an abstraction layer called "stickers." These are concise, structured summaries of key conditions and strategies that allow the model to utilize historical experience without being overwhelmed by context length.

Methodology: The Sticker-Driven Framework

The Sticker-TTS framework operates through a Recursive Reasoning Loop involving three specialized modules:

  1. Sticker Extractor ($E$): Distills a reasoning trace into a structured "sticker" containing key conditions and strategies.
  2. Sticker Modifier ($M$): Acts as a debugger, refining the sticker to correct logical flaws or missing information.
  3. Sticker Utilizer ($U$): Generates a new, improved response based on the refined sticker and the original question.
The Sticker-TTS framework architecture, featuring the recursive reasoning loop and the two-stage training process.

Two-Stage Optimization

To optimize these modules, the framework employs a two-stage training strategy:

  • Stage 1: Knowledge Distillation: Models are initialized using high-quality traces and stickers generated by powerful teacher models (e.g., DeepSeek-R1).
  • Stage 2: Self-improvement Bootstrapping: The framework iteratively practices on datasets, specifically filtering for "Correction Significance." This ensures the model learns the critical skill of transitioning from an incorrect initial response to a correct one using sticker guidance.

Evaluation and Results

Sticker-TTS was evaluated on elite mathematical benchmarks, including AIME 2024, AIME 2025, and Olym-MATH. The results indicate that the sticker-driven approach provides a more efficient scaling path than standard majority voting.

Comparative performance of Sticker-TTS (32B) against state-of-the-art models on mathematical reasoning benchmarks.

Key findings include:

  • Superior Accuracy: On AIME 2025, the 7B version of Sticker-TTS achieved a 12.42% relative improvement over standard self-consistency.
  • Competitive Performance: The 32B model consistently outperformed reinforcement-learning-tuned models like Skywork-OR1 and Light-R1 across multiple benchmarks.
  • Efficiency: Sticker-TTS achieved higher accuracy with 10 iterations than many baselines achieved with 64 independent samples, demonstrating the power of historical experience utilization.

Discussion and Insights

The success of Sticker-TTS highlights the importance of abstraction in reasoning. By forcing the model to condense its thoughts into "stickers," the framework prevents the "forgetting" issue common in long-context windows and prioritizes error recovery. While currently focused on mathematics, the principles of iterative refinement and diagnostic summaries offer a promising path for scaling reasoning capabilities in other complex domains.

Summary updated

Abstract

Large reasoning models (LRMs) have exhibited strong performance on complex reasoning tasks, with further gains achievable through increased computational budgets at inference.However, current test-time scaling methods predominantly rely on redundant sampling, ignoring the historical experience utilization, thereby limiting computational efficiency.To overcome this limitation, we propose Sticker-TTS, a novel test-time scaling framework that coordinates three collaborative LRMs to iteratively explore and refine solutions guided by historical attempts.At the core of our framework are distilled key conditions-termed stickers-which drive the extraction, refinement, and reuse of critical information across multiple rounds of reasoning.To further enhance the efficiency and performance of our framework, we introduce a two-stage optimization strategy that combines imitation learning with self-improvement, enabling progressive refinement.Extensive evaluations on three challenging mathematical reasoning benchmarks, including AIME-24, AIME-25, and Olym-MATH, demonstrate that Sticker-TTS consistently surpasses strong baselines, including self-consistency and advanced reinforcement learning approaches, under comparable inference budgets.These results highlight the effectiveness of sticker-guided historical experience utilization.
BibTeX
@misc{chen2025sticker,
  author = {Jie Chen and Jianping Jiang and Yingqian Min and Zican Dong and Shijie Wang and Xin Zhao and Ji-Rong Wen},
  title = {{Sticker-TTS: Learn to Utilize Historical Experience with a Sticker-driven Test-Time Scaling Framework}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.621},
}
Figure