Embedded Mean Field Reinforcement Learning for Perimeter-defense Game

Li Wang, Xin Yu, Xuxin Lv, Gangzheng Ai, Wenjun WuarXiv 2025 · DOI · arXiv

Abstract

This paper presents a novel approach to large-scale, 3D perimeter defense using Embedded Mean-Field Actor-Critic (EMFAC). Perimeter defense involves a team of defenders intercepting attackers before they breach a protected boundary. While traditional methods struggle with high dimensionality, scalability, and agent heterogeneity, EMFAC leverages representation learning and reward-based attention to coordinate dozens of diverse agents. We derive the Nash Equilibrium for 3D scenarios and demonstrate through simulations and real-world UAV experiments that EMFAC achieves superior performance, faster convergence, and robust transferability.

Introduction

Perimeter defense is a classic pursuit-evasion problem where attackers aim to reach a protected region, such as a base or a ship, while defenders attempt to intercept them. Moving this problem from 2D planes to 3D space introduces significant complexity. Furthermore, as the number of agents increases, traditional Multi-Agent Reinforcement Learning (MARL) faces an exponential explosion in state space. Real-world applications are further complicated by agent heterogeneity—different drones or missiles may have varying dynamics, such as velocity-controlled versus acceleration-controlled systems.

3D Perimeter Defense Scenario: Attackers (orange) attempt to breach the hemispherical protection boundary while defenders (blue) intercept them.

This work addresses these challenges by combining game-theoretic foundations with a scalable reinforcement learning framework designed for complex, wind-perturbed 3D environments.

Theoretical Foundation

The researchers established a baseline by deriving the Nash Equilibrium (NE) for a 3D hemispherical defense scenario. A key contribution is the proof that the optimal 3D strategy can be reduced to a 2D geometric problem through coordinate transformations. This reduction reveals that the optimal breach point remains invariant even as agents move at maximum speeds, provided they maintain their optimal trajectories. This theoretical grounding provides a "gold standard" for agent behavior in noise-free environments and informs the design of the learning-based approach.

The EMFAC Framework

To handle the complexities of large-scale, heterogeneous agent coordination, the authors proposed the Embedded Mean-Field Actor-Critic (EMFAC) framework. EMFAC addresses scalability and diversity through three primary innovations:

  1. Mean-Field Reinforcement Learning: Instead of tracking every individual agent's state, which is computationally prohibitive, each agent reacts to the average (mean field) effect of its neighbors. This significantly reduces the dimensionality of the observation space.
  2. High-level Action Encoder: To manage heterogeneity, representation learning is used to map diverse raw control signals into a unified "high-level action" space. By training an encoder to predict the next state, the model learns to focus on the intent of an agent's movement rather than its specific mechanical dynamics.
  3. Agent-level Attention: Not all neighboring agents are equally relevant to a defender's decision-making. EMFAC employs a lightweight attention mechanism, guided by reward representations, to prioritize agents that are most likely to impact the mission—such as those on a collision course or nearing the perimeter.
The EMFAC Architecture: The framework includes representation learning for high-level actions and an agent-level attention module for efficient reinforcement learning.

Experimental Evaluation

The framework was rigorously tested across multiple benchmarks:

  • Custom 3D Simulation: Built on the Dec-POMDP framework, the simulation included 16 distinct dynamic types and stochastic wind perturbations to mimic real-world conditions.
  • Scalability: Scenarios ranged from 10v10 to 50v50 agents, testing the limits of the mean-field approximation.
  • Real-World Validation: A 2v2 experiment was conducted using Crazyflie UAVs in an indoor arena with a motion capture system. This test proved the algorithm's ability to handle real-world sensor noise and physical flight dynamics.

Key Results and Insights

EMFAC consistently outperformed established baselines, including MADDPG, MATD3, and MTMFAC. In large-scale 50v50 tasks, it maintained a significantly higher success rate and a lower collision rate than its competitors.

The high-level action embeddings proved crucial for faster convergence; by concentrating the action space, the Q-network could learn more efficiently. Furthermore, the reward-guided attention mechanism was more effective than standard spatial attention, as it prioritized agents based on their strategic importance rather than just proximity. The real-world UAV tests confirmed that EMFAC is physically feasible and robust enough for deployment on actual hardware.

Conclusion and Future Work

EMFAC represents a significant step toward practical, large-scale 3D perimeter defense. By abstracting agent dynamics and focusing on mean-field interactions, the framework achieves a balance between theoretical optimality and computational efficiency. Future research will address scenarios with asymmetric agent counts (e.g., swarm attacks) and incorporate varying priority levels for different types of threats.

Summary updated

BibTeX
@misc{wang2025embedded,
  author = {Li Wang and Xin Yu and Xuxin Lv and Gangzheng Ai and Wenjun Wu},
  title = {{Embedded Mean Field Reinforcement Learning for Perimeter-defense Game}},
  year = {2025},
  eprint = {2505.14209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.14209},
}
Figure