RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework

Hao Gao, Shaoyu Chen, Yifan Zhu, Yuehao Song, Wenyu Liu, Qian Zhang, Xinggang WangarXiv 2026 · arXiv

TL;DR

RAD-2 is a unified generator-discriminator framework that stabilizes reinforcement learning (RL) for autonomous motion planning. By decoupling trajectory generation (via a diffusion model) from evaluation (via an RL-trained discriminator) and utilizing a novel high-throughput "BEV-Warp" simulator, the system achieves a 56% reduction in collision rates compared to state-of-the-art diffusion planners.

What Problem Does This Paper Address?

High-level autonomous driving requires planners that can handle multimodal future uncertainties—like deciding whether to overtake a slow car or wait behind it—while remaining robust in interactive, closed-loop environments.

Current diffusion-based planners are excellent at modeling these complex, multimodal trajectory distributions. However, they face two major hurdles:

  1. Training Instability: When trained purely on imitation learning (IL), they lack corrective feedback for "bad" behaviors not seen in expert data.
  2. RL Complexity: Applying RL directly to high-dimensional trajectory spaces is notoriously difficult. Sparse scalar rewards (like "did you crash?") provide a weak signal for fine-tuning complex curves, leading to severe credit assignment problems and unstable training.

Furthermore, existing simulators for RL training are often too slow (photorealistic reconstruction), too simple (game engines with unrealistic actors), or prone to temporal drift (generative world models).

Methods

The authors propose a synergistic architecture that splits the planning task into two specialized components:

  • Diffusion-based Generator: This module produces a diverse set of candidate trajectories. It is pre-trained on massive amounts of human driving data to capture realistic motion priors.
  • RL-based Discriminator: Instead of the generator learning from raw rewards, a Transformer-based discriminator is trained via RL to rerank the generator's candidates. It learns to assign higher scores to trajectories that lead to safe, efficient long-term outcomes.
Comparison of planning paradigms. RAD-2 (c) synergizes generation and selection in a closed-loop loop.

To make this work, they introduced two key algorithmic innovations:

  1. Temporally Consistent Group Relative Policy Optimization (TC-GRPO): This stabilizes RL by enforcing "latched execution"—reusing a selected trajectory for a short horizon. This ensures that rewards are attributed to consistent behavioral intentions rather than high-frequency noise.
  2. On-policy Generator Optimization (OGO): To prevent the generator from becoming a bottleneck, OGO uses closed-loop feedback to shift the generator's distribution toward high-reward manifolds using structured longitudinal signals (e.g., slowing down if a collision is imminent).

Finally, they developed BEV-Warp, a simulator that performs closed-loop evaluation by spatially warping Bird’s-Eye View (BEV) features based on the ego-vehicle's motion. This bypasses expensive image rendering, enabling high-throughput training while maintaining spatial consistency.

Schematic of the BEV-Warp simulation environment.

Datasets

  • Pre-training: 50,000 hours of real-world driving logs.
  • Closed-loop Training: 50,000 clips (10–20 seconds each) extracted from real-world logs, categorized into safety-critical and efficiency-oriented scenarios.
  • Benchmarks: The system was evaluated on the Senna-2 benchmark, which includes both open-loop scenarios and a photorealistic 3D Gaussian Splatting (3DGS) closed-loop environment.

Key Insights and Ideas

  • Decoupling RL from High-Dim Space: The most significant "aha moment" is that RL doesn't need to touch the high-dimensional trajectory coordinates directly. By training the discriminator on a low-dimensional score space, the optimization becomes much more tractable.
  • Temporal Coherence as a Physical Prior: The authors argue that frequent mode-switching in RL rollouts dilutes the reward signal. Enforcing temporal consistency (latched execution) acts as a physical prior that denoises the policy gradient.
  • Feature-Level Simulation: You don't need to render pixels to train a planner. If your perception system works in BEV, warping those BEV features is a highly efficient way to simulate "what if" scenarios without the overhead of a full game engine or generative world model.

Key Results

RAD-2 demonstrates significant gains across safety and efficiency metrics:

  • Safety: In safety-oriented scenarios, it reduced the collision rate (CR) from 0.533 (ResAD baseline) to 0.234, a 56% improvement. The at-fault collision rate dropped even further, from 0.264 to 0.092.
  • Efficiency: Navigational efficiency (Ego Progress) improved, with the model successfully completing 100% of its goals in 73.6% of scenarios, compared to 51.6% for the baseline.
  • Qualitative: Qualitative results show RAD-2 performing proactive decelerations to avoid collisions that baselines missed, and executing agile lane changes to bypass slow-moving traffic.
Qualitative comparison showing RAD-2 performing proactive deceleration to avoid a collision.

Limitations and Future Work

  • Representation Specificity: BEV-Warp is currently tied to BEV-centric perception architectures. It wouldn't work out-of-the-box for end-to-end models that use raw camera pixels or unified latent embeddings without explicit spatial grids.
  • Transition to Generative World Models: While BEV-Warp is efficient, it lacks the photorealism and flexibility of full world models. The authors suggest that future work will focus on improving the inference efficiency and temporal consistency of latent-based world models to replace feature-level warping.

Summary updated

Abstract

High-level autonomous driving requires motion planners capable of modeling multimodal future uncertainties while remaining robust in closed-loop interactions. Although diffusion-based planners are effective at modeling complex trajectory distributions, they often suffer from stochastic instabilities and the lack of corrective negative feedback when trained purely with imitation learning. To address these issues, we propose RAD-2, a unified generator-discriminator framework for closed-loop planning. Specifically, a diffusion-based generator is used to produce diverse trajectory candidates, while an RL-optimized discriminator reranks these candidates according to their long-term driving quality. This decoupled design avoids directly applying sparse scalar rewards to the full high-dimensional trajectory space, thereby improving optimization stability. To further enhance reinforcement learning, we introduce Temporally Consistent Group Relative Policy Optimization, which exploits temporal coherence to alleviate the credit assignment problem. In addition, we propose On-policy Generator Optimization, which converts closed-loop feedback into structured longitudinal optimization signals and progressively shifts the generator toward high-reward trajectory manifolds. To support efficient large-scale training, we introduce BEV-Warp, a high-throughput simulation environment that performs closed-loop evaluation directly in Bird's-Eye View feature space via spatial warping. RAD-2 reduces the collision rate by 56% compared with strong diffusion-based planners. Real-world deployment further demonstrates improved perceived safety and driving smoothness in complex urban traffic.
BibTeX
@misc{gao2026rad,
  author = {Hao Gao and Shaoyu Chen and Yifan Zhu and Yuehao Song and Wenyu Liu and Qian Zhang and Xinggang Wang},
  title = {{RAD-2: Scaling Reinforcement Learning in a Generator-Discriminator Framework}},
  year = {2026},
  eprint = {2604.15308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.15308},
}

Loading related content...

Figure