BIRD-RL: Scaling Agentic Reinforcement Learning over Stateful Data-Centric Environments

Ge Qu, Jinyang Li, Xiaohan Xu, Xiaolong Li, Bowen Qin, Nan Huo, Ziwei Tang, Chenhao Ma, Reynold ChengarXiv 2026 · arXiv

TL;DR

BIRD-RL is a new framework designed to train language agents for complex, stateful database tasks like SQL debugging. By combining a four-stage reinforcement learning (RL) curriculum with a novel "trajectory-scoped" execution infrastructure, it achieves performance competitive with proprietary models like Claude-3-Opus while being 12.8x more I/O efficient than standard container-per-rollout approaches.

What Problem Does This Paper Address?

Most current LLM-based agents excel at "one-shot" tasks like generating a single SQL query from natural language. However, real-world database work is stateful and iterative. Debugging a broken SQL script often requires making changes (UPDATE, DELETE, ALTER TABLE), checking the results, and refining the approach.

Existing RL frameworks are "tool-centric"—they treat the environment as a stateless black box. To prevent "trajectory contamination" (where one agent's database changes interfere with another's), researchers typically spin up a fresh Docker container for every single rollout. For large-scale RL training involving thousands of iterations, this creates a massive I/O bottleneck and makes training prohibitively expensive and slow.

Methods

The authors introduce two major innovations to bridge this gap:

  1. A Four-Stage RL Curriculum: Rather than training end-to-end immediately, which is often unstable, BIRD-RL uses a progressive pipeline:
  2. Stage 1 (Reasoning Imitation): Supervised fine-tuning (SFT) on "thoughts" to establish a reasoning prior.
  3. Stage 2 (Tool-Use Imitation): SFT on multi-turn trajectories to teach the agent how to use database tools.
  4. Stage 3 (Adaptive Reasoning RL): Using Group Relative Policy Optimization (GRPO) to refine reasoning before the agent starts interacting with the live DB.
  5. Stage 4 (Stateful Interactive Agentic RL): Full on-policy RL where the agent explores the environment and receives rewards based on final success.
  6. Trajectory-scoped Persistent Execution Infrastructure: To solve the scaling problem, they built a system that manages a pool of database replicas.
  7. Dirty-Aware Lazy Reset: Instead of nuking a container after every turn, the system tracks if a database was actually modified. If not, it’s reused immediately. If it was "dirtied," the reset is deferred until the replica is needed again, avoiding lock contention.
  8. Worker-Partitioned Pool: Databases are pre-allocated to specific workers to ensure high concurrency without the overhead of Kubernetes.
Figure 1: Overview of the BIRD-RL framework. (Left) The four-stage agentic RL curriculum transitioning from static reasoning imitation to interactive stateful exploration. (Right) The Trajectory- Scoped Persistent Execution Infrastructure, enabling scalable and isolated database replica reuse via a customized agent loop and worker-partitioned pool.

Datasets

  • SIX-GYM-SQLITE: A newly introduced training environment containing 5,000 high-quality instances adapted from Stack Overflow and other sources, specifically for SQLite.
  • BIRD-CRITIC-SQLite: The primary evaluation benchmark, featuring 500 complex SQLite issues across query, management, and personalization categories.
  • BIRD-SQL: Used to evaluate the model's ability to still perform standard text-to-SQL generation.

Key Insights and Ideas

  • Imitation is a Floor, Not a Ceiling: SFT alone (Stage 1 & 2) often results in "rigid" models that can't adapt when their first guess fails. The RL stages (3 & 4) are critical for teaching the agent to actually use execution feedback to pivot its strategy.
  • The "Statefulness" Tax is Real: The authors found that only ~9.25% of trajectories actually modify the database. By identifying these "clean" trajectories, their infrastructure avoids thousands of unnecessary database resets.
  • Unified Models are Possible: They trained a model called BIRD-ZENO that handles both debugging and generation. They discovered a trade-off: debugging data helps exploration, while generation data helps schema grounding. A 3:1 (debug:gen) ratio was the "sweet spot."

Key Results

The BIRD-TALON model family (based on Qwen-2.5-Coder) shows impressive gains:

  • Performance: The 14B model achieved a 48.00% Success Rate on BIRD-CRITIC-SQLite, effectively matching the performance of the much larger Claude-3-Opus Agent (48.20%).
  • Efficiency: Their infrastructure reduced measured online lifecycle I/O by 12.8x compared to the standard per-rollout container approach.
Table 1: Success Rate (SR %) of different models on BIRD-CRITIC-SQLITE, evaluated under greedy decoding and grouped by issue category. Bold numbers indicate the highest score in each column, and underlined numbers indicate the second highest. "Quer." refers to query-like issues, "Mana." refers to data-management issues, and "Pers." refers to personalized-function issues.
Table 3: Comparison of database lifecycle I/O during interactive rollout execution under trajectory- level isolation. “Emp. I/O” reports the measured online database lifecycle I/O from our implementa- tion and excludes the one-time offline pool materialization cost.

Limitations and Future Work

  • Dialect Focus: While the paper shows some generalization to other SQL dialects (PostgreSQL, MySQL), the core training was done on SQLite.
  • Reward Hacking: The authors noted that complex "proxy rewards" (like scoring intermediate steps) often led to reward hacking. They stuck to a simple binary "did it work?" reward, but suggest that more robust, semantics-aware rewards are a key area for future research.
  • Scope: The framework is currently focused on SQL, but the authors believe the "trajectory-scoped" infrastructure could be extended to other stateful data science tasks like data cleaning and automated workflow construction.

Summary updated

Abstract

While agentic reinforcement learning (RL) has emerged as a promising paradigm for training small language models (SLMs), existing frameworks are predominantly tool-centric and struggle with data-centric environments. Specifically, applying tool-centric frameworks to stateful tasks like SQL debugging introduces severe spatial inefficiencies since proper environment isolation typically requires a separate database container per rollout. Naively reusing databases without tracking evolving states across multi-turn interactions causes trajectory contamination and cross-trajectory interference, which severely destabilizes policy updates. To overcome these bottlenecks, we introduce BIRD-RL, a novel four-stage agentic RL curriculum that alternately enhances foundational agent reasoning and trajectory-adaptive on-policy exploration to circumvent the instability inherently associated with direct end-to-end learning. Crucially, to support this curriculum, BIRD-RL is driven by a Trajectory-scoped Persistent Execution Infrastructure, which enables safe database replica reuse while preserving trajectory isolation. This design improves spatial RL efficiency by 12.8x, successfully enabling scalable agentic RL for complex data-centric tasks without needs of Kubernettes. Empirical evaluations on the BIRD-CRITIC-SQLite benchmark demonstrate that our 7B and 14B models achieve Success Rates of 44.40% and 48.00%, respectively, showing competitive performance against strong proprietary model-based agents such as Claude-Opus-4.6 Agent (48.20%). Furthermore, BIRD-RL successfully trains unified SLMs capable of both advanced SQL debugging and generation, highlighting its broader potential for multi-objective optimization across heterogeneous, real-world data-centric tasks.
BibTeX
@misc{qu2026bird,
  author = {Ge Qu and Jinyang Li and Xiaohan Xu and Xiaolong Li and Bowen Qin and Nan Huo and Ziwei Tang and Chenhao Ma and Reynold Cheng},
  title = {{BIRD-RL: Scaling Agentic Reinforcement Learning over Stateful Data-Centric Environments}},
  year = {2026},
  eprint = {2610.29426},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.29426},
}

Loading related content...

Figure