AutoRL: A Multi-Agent System towards Autonomous Reinforcement Learning

Simon Tang, Wenjie Shi, Yongjing Yin, Keheng Wang, Jialin Liu, Zheng Zhang, Linvo Lin, Xiangyuan Xue, Kan Ren, Jingang Wang, Xunliang CaiarXiv 2026 · arXiv

TL;DR

AutoRL is a multi-agent system designed to act as an autonomous reinforcement learning (RL) researcher, capable of monitoring, diagnosing, and intervening in training runs in real-time. By automating the "babysitting" of unstable RL processes, it enables long-term, unattended model improvement and failure recovery, marking a significant step toward recursive self-improvement (RSI) in AI.

What Problem Does This Paper Address?

Reinforcement learning is a cornerstone of modern AI post-training (e.g., for reasoning and coding models), but it is notoriously difficult to automate. RL training is often unstable: gradients can explode, models can "stagnate" on performance plateaus, or reward functions can be "hacked" by the model.

Currently, human researchers must constantly monitor training logs to decide when to adjust hyperparameters or fix bugs. While some existing systems automate the sequence of experiments (run an experiment, analyze, then start a new one), they lack the ability to intervene during a run. This "process-level" supervision is what AutoRL aims to automate, solving the challenges of timely intervention and the high computational cost of restarting failed experiments.

Methods

AutoRL mimics the workflow of a human researcher using a multi-agent framework with three specialized roles:

  1. Perceptors (The Eyes): Four specialized agents monitor different signals—training stability (gradients/loss), rollout behavior (rewards), evaluation performance, and system efficiency. They trigger "deliberation" only when they detect anomalies or stagnation.
  2. Proposers (The Brains): When a signal is triggered, three agents—the Tuner (refines hyperparameters), the Innovator (explores new mechanisms), and the Reviewer (checks code correctness)—independently diagnose the issue. They then engage in a multi-round deliberation to rank and merge candidate interventions.
  3. Orchestrator (The Manager): This agent reviews the final recommendation and decides how to apply it. It uses two primary mechanisms:
  4. HOT Updates: Online modifications (like adjusting an entropy coefficient) applied without stopping the training.
  5. COLD Resumptions: Stopping the run, applying a code patch or data change, and resuming from a compatible checkpoint.
The AutoRL framework separates the training lifecycle into management, continuous perception, and improvement research.

Datasets

The authors evaluated AutoRL across two primary domains:

  • Mathematical Reasoning: Training Qwen3 (1.7B and 4B) and LongCat-Flash-Lite on the DAPO-Math-17K dataset. Evaluation used high-stakes benchmarks like AIME 2024/2025, HMMT 2025, Beyond AIME, and IMO-AnswerBench.
  • Agentic Coding: A long-term (14.7 day) experiment using SWE-related tasks to test the system's ability to handle real-world software engineering complexities and runtime failures.
  • General Knowledge: Evaluated using MMLU-Pro and GPQA-Diamond to ensure improvements in reasoning didn't come at the cost of general capabilities.

Key Insights and Ideas

  • Monitoring vs. Research Rhythms: The authors found that monitoring requires frequent, shallow checks, while research requires deep, sustained investigation. Separating these into different agent roles (Perceptors vs. Proposers) prevents execution output from "crowding out" critical analysis.
  • Inexpensive Validation: Before committing expensive GPU resources to a new hypothesis, AutoRL agents perform "CPU-only" checks—running small-scale tests or replaying stored trajectories—to filter out implementation errors.
  • The "Flywheel" of RSI: AutoRL demonstrates that an AI can not only improve its own performance but also maintain the infrastructure of its own improvement, such as fixing its own training code or recovering from NaN (Not-a-Number) numerical failures.
  • HOT vs. COLD Interventions: The ability to apply "HOT" updates allows the system to pivot training strategy without the overhead of a full restart, significantly increasing the "distinct training steps" reached within a fixed compute budget.

Key Results

AutoRL consistently outperformed both static RL recipes and a general-purpose agent baseline:

  • Performance Gains: On Qwen3-4B, AutoRL improved AIME 2024 accuracy from 46.67% to 66.67% (+20%) compared to a static recipe.
  • Unattended Reliability: In a 14.7-day coding run, AutoRL autonomously fixed 11 runtime failures (including NaNs and communication errors). It increased pass@1 on SWE tasks by 12.84% without human intervention.
  • Efficiency: AutoRL reached its peak performance while spending 0% of its budget on "repeated steps" (revisiting old indices due to rollbacks), whereas a general agent spent nearly 45% of its budget on such repetitions.
Figure 1: AutoRL: autonomous RL supervision and research. Right: AutoRL continually adopts interventions, such as fixes and adjustments, to sustain gains over a static recipe.

Limitations and Future Work

  • Incremental vs. Fundamental Innovation: While AutoRL is excellent at "local refinements" (tuning, repairing, and reward shaping), the authors admit it struggled to produce "fundamental methodological breakthroughs"—it didn't invent entirely new RL algorithms from scratch.
  • Backbone Dependency: The system's effectiveness is tied to the strength of the underlying LLM agents. Using a weaker backbone (like GLM-5.2 vs. GPT-6.1) led to a narrower research scope and lower peak accuracy.
  • Future Directions: The authors suggest that the next frontier for RSI lies in developing agents that can propose and validate more fundamental algorithmic changes, which currently remains a high-risk, resource-intensive task.

Summary updated

Abstract

Recursive self-improvement (RSI) is one of the most promising paths toward an AI singularity for superintelligence. Achieving this vision requires AI systems to undertake the full training pipeline for iterative improvement. Reinforcement learning (RL) plays an essential role in this cycle to improve model capabilities and generalization, yet automating RL remains difficult due to unstable training dynamics and the need for sustained supervision. We introduce AutoRL, a multi-agent system for automating RL training, as a cornerstone for future RSI. Different from conducting research by end-to-end complete experiments, AutoRL monitors the evolving training state and integrates process-level supervision and research into an ongoing RL training, following the practical workflow of an RL researcher. Specifically, AutoRL consists of three RL-specific agentic roles. Perceptors track training dynamics, rollouts, evaluation, and efficiency, triggering deliberation when they detect anomalies, degradation, or stagnation. Three complementary Proposers diagnose problems and refine candidate interventions through shared evidence and checks. The Orchestrator reviews their recommendation and applies approved changes through online hot updates or checkpoint-based resumption. Experiments on Qwen3 and LongCat-Flash-Lite demonstrate the superiority of AutoRL over static RL recipes and a general agent. The research records show that existing agents can handle unattended RL supervision and dynamic adjustment. The difficulty for future RSI lies in developing fundamental methods that are valuable for resource-consuming validation.
BibTeX
@misc{tang2026autorl,
  author = {Simon Tang and Wenjie Shi and Yongjing Yin and Keheng Wang and Jialin Liu and Zheng Zhang and Linvo Lin and Xiangyuan Xue and Kan Ren and Jingang Wang and Xunliang Cai},
  title = {{AutoRL: A Multi-Agent System towards Autonomous Reinforcement Learning}},
  year = {2026},
  eprint = {2611.17790},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17790},
}
Figure