RSI-Prime: Training Models to Train Models

Dewu Zheng, Yang Ye, Hongyu Zhang, Jun Yu, Yanlin WangarXiv 2026 · arXiv

TL;DR

RSI-Prime introduces a novel dataset and training methodology designed to teach Large Language Models (LLMs) how to autonomously conduct model-training experiments. By employing fine-grained, decision-level supervision—masking out poor choices even within successful experiments—the researchers achieved a 31.95% relative improvement in autonomous post-training performance, marking a significant step toward recursive self-improvement (RSI).

What Problem Does This Paper Address?

The "holy grail" of AI research is Recursive Self-Improvement (RSI): the idea of a model that can autonomously design, execute, and learn from experiments to create a better version of itself. While current LLMs are proficient at coding and basic planning, the specific domain of model training is incredibly complex. It requires long-horizon reasoning, data curation, hyperparameter tuning, and the ability to iterate based on evaluation feedback.

Existing approaches often treat training as a "black box" or rely on simple imitation of successful trajectories. However, a successful experiment doesn't mean every decision made during that experiment was optimal. The authors identify a critical gap: LLMs need to learn from the context of training practice while being shielded from imitating the low-quality "noise" that inevitably occurs during complex, multi-step experimental workflows.

Three paradigms of model training: manual, model-assisted, and fully automated.

Methods

The authors developed a systematic pipeline to move from raw experimental data to high-quality supervision:

  1. Trajectory Collection & Validation: They built a robust environment where agents conduct training experiments. To ensure data quality, they implemented "pre-execution validation" to check protocols, scorers, and runtimes before any data was collected.
  2. Fine-Grained Decision Identification: Instead of viewing a trajectory as a simple sequence of messages, they broke it down into discrete decisions. Each decision consists of three parts: Analysis (the rationale), Choice (the intended action), and Execution (the actual tool call and result).
  3. Decision-Level Masked Training: This is the core innovation. They annotated the quality of every decision. During Supervised Fine-Tuning (SFT), they applied a "conservative masking rule": if an assistant message is linked to even one problematic decision, the entire message is masked from the loss function.
  4. Preserving Context: Critically, while the loss is masked for bad decisions, the content remains in the model's context window. This allows the model to see the "wrong" path and the subsequent corrections without being forced to imitate the error.
Overview of the RSI-Prime pipeline, from task validation to decision-level masked training.

Datasets

  • RSI-Prime (New): A dataset comprising 350 complete experimental trajectories (yielding 385 training segments) across seven categories, including mathematics, code, tools, and instruction following. It contains over 90,000 annotated decision records.
  • PostTrainBench: The primary evaluation benchmark. It assesses an agent's ability to improve a target model (in this case, Qwen3-4B-Base) under a fixed 10-hour execution budget.

Key Insights and Ideas

  • Success $\neq$ Quality: The authors found that even in experiments that met their "success" threshold (improving the target model significantly), about 7.5% of individual decisions were still problematic. Training on everything ("Full SFT") actually hurts performance compared to selective supervision.
  • The "Conservative Rule" Works: Masking entire messages associated with any bad decision (retaining ~75% of tokens) proved more effective than masking at a finer "step" level. This suggests that the coherence of a single turn of dialogue is important for learning.
  • Action-Specific Difficulty: Behavioral analysis revealed that agents struggle most with "implementation optimization" and "data source selection" (see Figure 3). These are the areas where high-quality supervision provides the most leverage.
Action taxonomy and decision quality distribution within the RSI-Prime dataset.

Key Results

The researchers fine-tuned GLM-5.3 using RSI-Prime and evaluated it on PostTrainBench:

  • Performance Leap: The "Decision SFT" model raised the overall score from 36.56 to 48.24, a 31.95% relative gain.
  • Superiority of Selective Supervision: Decision-level masking outperformed "Full SFT" (which scored 45.61) and other baselines like tool-failure masking.
  • Broad Behavioral Improvement: The model didn't just get lucky; its "quality pass rate" across 20 different action types improved from 87.25% to 96.11%.
  • Competitive Standing: The resulting open-source agent outperformed several prominent closed-source models, including Gemini 4 Argon and GPT-6.1 (Sol) on the same benchmark.
Comparison of decision quality across different supervision strategies, showing broad improvements for Decision SFT.

Limitations and Future Work

  • Data Overlap: The authors acknowledge a training-evaluation overlap for the HumanEval task within the benchmark, which is a common challenge in LLM research.
  • Supervisor Dependency: The current pipeline relies on an "independent supervisor" (another model) to monitor and recover trajectories. Future work could explore making the agent more self-reliant in handling environment failures.
  • Scaling RSI: While this paper focuses on the "training to train" step, the ultimate goal is a closed-loop system where the improved agent generates even better training data for the next generation, a cycle not fully explored here.

Summary updated

Abstract

Enabling large language models (LLMs) to autonomously conduct training and improve target models is an important foundation for recursive self-improvement (RSI). As LLMs expand their role from being trained to conducting training, their model-training capability warrants dedicated study and systematic development. To this end, we construct RSI-Prime, a dataset for improving model-training capability, comprising 350 complete experimental trajectories across seven task categories. To help LLMs learn to make high-quality experimental decisions from training practice, we perform fine-grained decision identification and quality annotation, then apply decision-level loss masking to exclude supervision on low-quality behavior. Experiments with GLM-5.3 on PostTrainBench, a benchmark for autonomous post-training, show that training on RSI-Prime yields a 31.95% relative improvement in overall performance over the original GLM-5.3, highlighting a path toward future RSI systems in which models autonomously sustain their own advancement.
BibTeX
@misc{zheng2026rsi,
  author = {Dewu Zheng and Yang Ye and Hongyu Zhang and Jun Yu and Yanlin Wang},
  title = {{RSI-Prime: Training Models to Train Models}},
  year = {2026},
  eprint = {2611.23138},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23138},
}
Figure