Distilling Safety Filters for Filter-free Multi-agent RL

Srikar Babu Gadipudi, Somayeh Sojoudi, Claire J. TomlinarXiv 2026 · arXiv

TL;DR

This paper introduces a framework for training multi-agent reinforcement learning (MARL) policies that are guaranteed to be collision-free during training via a centralized safety filter, yet can be deployed as decentralized, "filter-free" policies. By combining Implicit Q-learning (IQL) with a novel "off-support" distillation technique, the authors create local-sensing policies that scale zero-shot to large teams while maintaining rigorous safety bounds.

What Problem Does This Paper Address?

Reinforcement Learning (RL) is notorious for its "trial-and-error" nature. In robotics, the "error" part often means a costly collision. To prevent this, researchers use Control Barrier Functions (CBFs) as safety filters. These filters act like a "guardian angel," minimally correcting a robot's proposed action to ensure it stays within a safe set.

However, two major problems arise:

  1. The Computational Bottleneck: In multi-agent systems, a centralized safety filter must solve a Quadratic Program (QP) that considers every agent simultaneously. The number of constraints grows quadratically ($O(n^2)$) with the team size. For a team of 20 robots, the filter might take longer to compute than the actual control cycle allows.
  2. The Training-Deployment Mismatch: If a robot is trained with a filter, it never "sees" the consequences of unsafe actions. Standard RL algorithms (like SAC) end up overvaluing unsafe actions because the filter masks their cost—a phenomenon called action aliasing. Furthermore, the robot's experience buffer lacks data on near-collision states because the filter prevents it from ever getting that close.

The authors seek to use the centralized filter as a "teacher" during training but discard it entirely at deployment, leaving a fast, decentralized policy that knows how to stay safe on its own.

Methods

The authors propose a two-pronged training strategy:

  • Implicit Q-Learning (IQL) for Policy Extraction: Instead of using standard actor-critic methods that might overvalue corrected actions, they use IQL. IQL learns strictly from the executed safe actions in the replay buffer. This ensures the policy doesn't learn to rely on a "guardian angel" that won't be there later.
  • Off-Support Safety-Filter Distillation (OSFD): Since the safety filter keeps the robots away from danger, the training data is naturally sparse near obstacles. OSFD solves this by generating synthetic near-collision states—mathematically placing agents in "scary" configurations they rarely visit—and querying the centralized filter for the correct safe response. The policy is then trained to mimic these labels.
  • Attention-Based Decentralized Architecture: The policy uses a cross-attention mechanism to process observations from nearby agents and obstacles. Because attention treats neighbors as "tokens," the policy's parameter count doesn't change regardless of how many agents are in the scene, enabling zero-shot scaling.
  • Theoretical Safety Guarantees: The paper provides a mathematical "chain of bounds." They prove that minimizing the distillation loss directly bounds the policy's "CBF residual" (how much it violates safety constraints), which in turn bounds the probability of a collision during deployment.

Datasets

The approach was evaluated using:

  • Simulated Environments: Custom 2D arenas (10x10m) with Single-Integrator (SI) and Double-Integrator (DI) dynamics. Scenarios included a "circle layout" (a stress test where all agents cross through the center) and random layouts.
  • Hardware: Experiments were conducted using Crazyflie 2.x quadrotors in a flight arena equipped with a Vicon motion capture system.

Key Insights and Ideas

  • The "Guardian Angel" Problem: The authors identify that a safety filter is a double-edged sword. It keeps you safe, but it also "hides" the very boundaries the RL agent needs to learn. You cannot learn to avoid a cliff if you are never allowed to see the edge.
  • Imaginary Danger is Better than Real Crashes: Since we can't let real robots collide, OSFD effectively lets the robots "dream" about collisions. By supervising the policy on synthetic states, the agent learns the "sharp" boundary of the safety filter without ever actually crashing during training.
  • Constant-Time Scaling: By moving the safety logic from a centralized QP (which slows down as agents are added) into a decentralized neural network (which takes constant time per agent), the system becomes viable for massive swarms.

Key Results

  • Zero Collisions: In 1,800 test episodes across various simulated settings, the proposed local-sensing policies recorded zero collisions.
  • Zero-Shot Scaling: A policy trained with only 3 agents was deployed on a team of 20 agents without retraining. It maintained a >99% goal-reaching rate and stayed collision-free for teams of up to 8 agents in high-stress scenarios.
  • Speedup: At 20 agents, the centralized safety filter took 16.9 ms to solve, while the proposed decentralized policy took only 0.6 ms per agent.
  • Hardware Success: The team successfully flew up to five Crazyflie quadrotors simultaneously using the distilled policy without any safety filter in the loop, resolving complex crossings using only local information.

Limitations and Future Work

  • Concentrability Assumption: The theoretical safety bounds rely on an "Assumption of Coverage," meaning the synthetic states must sufficiently cover the states the robot visits during deployment. If the robot enters a completely novel environment, the safety guarantees may degrade.
  • Sensing Requirements: The hardware experiments still relied on an external motion capture system to provide the "local observations." The authors note that future work should replace this with on-board sensing (e.g., cameras or LiDAR).
  • Scaling Limits: While the policy scales well, collisions did begin to appear (3.3%) when pushed to 20 agents in the most crowded "random" layouts, suggesting there is still a performance gap between local sensing and global optimization at extreme densities.

Summary updated

Abstract

Reinforcement learning (RL) can teach robot teams to navigate shared spaces, but trial and error causes collisions that real robots cannot afford. Control barrier function (CBF) safety filters prevent these collisions by correcting unsafe actions, yet in a team the filter becomes a centralized optimization whose number of constraints grows quadratically with the team size. We propose an approach that builds the filter's safety into the RL policy and examine how such a system behaves during filter-free deployment. However, the filter that keeps training safe also hides what the policy must learn: it masks the cost of unsafe actions, which actor-critic methods then overvalue, and it keeps near-collision states out of the training data. Our approach therefore learns only from executed safe actions with implicit Q-learning and distills the filter into the policy at synthetic near-collision states. We prove that filter distillation loss bounds the policy's constraint violation and hence how far agents can cross their safety margin, so errors never accumulate. When distillation states cover deployed states, the loss also bounds the collision probability at every step. Without a filter and acting only on local observations, our policies have no collisions in 1,800 test episodes with multiple agents. A policy trained with three agents transfers zero-shot to teams of twenty at constant per-agent computation and stays collision-free with up to eight agents. We test our policies on Crazyflie quadrotors with up to five flying at the same time.
BibTeX
@misc{gadipudi2026distilling,
  author = {Srikar Babu Gadipudi and Somayeh Sojoudi and Claire J. Tomlin},
  title = {{Distilling Safety Filters for Filter-free Multi-agent RL}},
  year = {2026},
  eprint = {2611.19190},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19190},
}
Figure