CoopDistill: Equilibrium-Guided Multi-Agent Policy Distillation for Cooperative Driving

Shiyu Fang, Sheng Zhao, Jiarui Zhang, Chengkai Xu, Xiaocong Zhao, Peng Hang, Jian SunarXiv 2026 · arXiv

Cooperative driving—where vehicles coordinate their movements to improve safety and flow—is a cornerstone of the future of transportation. However, current approaches usually fall into two camps: rigid optimization methods that are safe but inflexible, and reinforcement learning (RL) methods that are flexible but difficult to train and verify.

CoopDistill bridges this gap. It is a hybrid framework that uses a game-theoretic "Potential Game" (PG) as a safe reference and trains a multi-agent RL policy to learn clever "residuals" (adjustments) around that reference. By focusing on the hardest scenarios where traditional optimization fails, CoopDistill achieves significant gains in traffic efficiency without compromising on safety.

TL;DR

CoopDistill combines the safety of game-theoretic optimization with the flexibility of multi-agent reinforcement learning by learning adjustments to a safe reference policy. It uses a hard-case curriculum to master complex interactions and a rollout selector to ensure every move is safer and more efficient than the baseline, resulting in zero collisions and up to 27% less delay.

What Problem Does This Paper Address?

The authors identify three primary hurdles in integrating optimization-based methods with Multi-Agent Reinforcement Learning (MARL):

  1. The Exploration-Convergence Trade-off: Searching for the perfect joint maneuver for multiple vehicles across longitudinal and lateral paths creates a massive search space that is hard for RL to navigate.
  2. Insufficient Long-Tail Learning: Most traffic interactions are routine and easily solved by simple rules. If an RL agent spends all its time practicing these "easy" cases, it never learns how to handle the rare, complex "long-tail" scenarios where coordination is actually difficult.
  3. Execution-Time Reliability: RL policies are "black boxes" that don't explicitly enforce safety constraints. In a new, unseen situation, a learned policy might propose a move that causes a collision.
The three core challenges: Exploration-Convergence, Long-Tail Learning, and Execution Reliability.

Methods

CoopDistill introduces a multi-layered architecture to solve these problems:

  • Potential-Game (PG) Reference: The system starts with a PG formulation that provides a baseline for longitudinal coordination (who goes first) on fixed paths. This provides a "good enough" starting point for the RL.
  • Residual MARL Policy: Instead of learning how to drive from scratch, the MARL policy (using MAPPO) learns residuals. It proposes small adjustments to acceleration, timing, and—crucially—lateral path corrections. This allows the system to deviate from fixed lanes to bypass obstructions.
  • Hard-Case Curriculum: The authors rank training scenes based on how poorly the PG reference performs (e.g., scenes with high delay or collisions). The training process then oversamples these "hard cases," forcing the RL to learn maneuvers that the optimizer couldn't find.
  • Comparative Rollout Selector: At execution time, the system performs a "look-ahead" simulation (rollout) for both the PG reference and the MARL proposal. It only executes the MARL proposal if it is collision-free and results in better traffic utility than the reference. If not, it falls back to the safe PG baseline.
Figure 2: Overview of CoopDistill. A PG coordination reference guides residual MARL learning, hard cases direct training experience, and rollout comparison selects the executed joint decision.

Datasets

The framework was evaluated in three high-conflict simulated environments:

  1. Merging Area: Vehicles entering a highway from a ramp.
  2. Roundabout: Complex weaving and yielding.
  3. Unsignalized Intersection: Multi-way conflict resolution.

The authors tested these scenarios across four levels of Connected Automated Vehicle (CAV) penetration (25%, 50%, 75%, and 100%) to see how the system handles mixed traffic with human-driven vehicles.

Key Insights and Ideas

  • Spatial Flexibility is the "Aha!" Moment: Traditional optimization often fixes vehicle paths to simplify the math. CoopDistill shows that allowing the RL to learn lateral path adjustments (like nudging over to let someone merge) is the key to breaking deadlocks that trap fixed-path methods.
  • Residuals Shrink the Search Space: By learning adjustments to a reference rather than absolute actions, the RL agent doesn't have to "re-invent the wheel" for basic driving, allowing it to converge much faster on sophisticated cooperative behaviors.
  • Safety via Fallback: The comparative rollout selector acts as a "safety shield." It allows the system to benefit from the creativity of RL while maintaining the rigorous safety guarantees of optimization-based methods.

Key Results

CoopDistill was compared against six baselines, including traditional reservation systems (FCFS) and pure MARL approaches.

  • Success Rate: CoopDistill improved success rates by 2.1 to 27.1 percentage points over the PG baseline.
  • Efficiency: It reduced mean vehicle delay by 7.3% to 27.3%.
  • Safety: Most impressively, CoopDistill recorded zero CAV-involved collisions across all test scenarios, whereas pure MARL baselines (like MADQN) frequently caused accidents.
Success rates across different scenarios and CAV penetration levels.

Limitations and Future Work

While CoopDistill excels at resolving individual encounters, the authors note a few areas for growth:

  • Sustained Traffic: The current model focuses on specific "scenes." Future work needs to address continuous traffic flow and long-term bottlenecks.
  • Communication Realities: The paper assumes perfect communication between CAVs. Real-world factors like latency, packet loss, and sensor noise remain to be explored.
  • Human Predictability: The model relies on a specific driver model for human-driven vehicles; testing against more diverse or adversarial human behaviors would be a valuable next step.

For those interested in the implementation, the code is available at https://github.com/FanGShiYuu/CoopDistill.

Summary updated

Abstract

Cooperative driving extends autonomous driving from individual vehicle decisions to joint maneuvers that improve traffic safety and efficiency. Optimization-based methods impose safety constraints but often fix paths to simplify multi-vehicle planning, limiting spatial flexibility. Multi-agent reinforcement learning (MARL) explores alternatives through interaction, but coordinated learning and safe execution remain challenging. Their integration nevertheless faces challenges in exploration efficiency, experience allocation, and execution reliability. To address these challenges, this paper proposes CoopDistill, an equilibrium-guided multi-agent policy distillation framework. A potential-game (PG) formulation provides a fixed-path longitudinal reference, around which a residual MARL policy learns longitudinal acceleration, conflict-approach timing, and lateral path corrections. A hard-case curriculum concentrates experience on interactions that PG leaves unresolved or substantially delayed. A comparative rollout selector evaluates the joint consequences of learned corrections against PG before choosing the executed action. Experiments in merging, roundabout, and intersection scenarios compare CoopDistill with six baselines at four connected automated vehicle penetration levels. Relative to PG, success improves by 2.1--27.1 percentage points and delay decreases by 7.3--27.3\%, with no observed collisions. Code is available at https://github.com/FanGShiYuu/CoopDistill
BibTeX
@misc{fang2026coopdistill,
  author = {Shiyu Fang and Sheng Zhao and Jiarui Zhang and Chengkai Xu and Xiaocong Zhao and Peng Hang and Jian Sun},
  title = {{CoopDistill: Equilibrium-Guided Multi-Agent Policy Distillation for Cooperative Driving}},
  year = {2026},
  eprint = {2611.21069},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21069},
}
Figure