Introduction
Deploying a fleet of autonomous drones for tasks such as disaster response or environmental monitoring presents significant logistical challenges. These agents typically operate under partial observability, where they can only sense their immediate surroundings, and limited communication range, restricting coordination to nearby neighbors.
Traditional path-planning algorithms, including TSP solvers and greedy heuristics, often struggle in dynamic environments where goal locations are not known a priori. While Deep Reinforcement Learning (DRL) offers a potential solution, standard architectures frequently fail to scale as the number of agents increases and struggle to maintain effective coordination under communication constraints. This research introduces a decentralized yet "cooperation-aware" decision-making engine that bridges this gap by combining Graph Neural Networks (GNNs) and Transformers with DRL.
Methodology
The proposed framework shifts from traditional convolutional processing of grid maps to a graph-centric world view. The architecture consists of several integrated stages designed to optimize multi-agent navigation:
- Adaptive Graph Construction: The environment is modeled as a dynamic graph where agents and discovered goals serve as nodes. Edges are established based on proximity and a $k$-nearest neighbor constraint to simulate realistic communication bandwidth limits.
- GNN Feature Extraction: A GNN updates node embeddings by aggregating local observations and semantic data from neighbors, effectively allowing agents to share perspectives.
- Transformer-Based Message Passing: A transformer layer with multi-head attention captures long-range dependencies across the graph. This allows the system to globally weigh the importance of specific agent-goal interactions.
- Double DQN with PER: These refined embeddings inform a Double Deep Q-Network (Double DQN) with Prioritized Experience Replay (PER), which selects discrete actions (Up, Down, Left, Right) to maximize long-term rewards.
Experimental Setup
The framework was evaluated using a custom environment built on the MiniGrid library. Testing spanned six configurations of increasing complexity, from a $10 \times 10$ grid with 2 agents and 10 goals to a $60 \times 60$ grid with 33 agents and 169 goals. The proposed model was benchmarked against baseline DQN, Particle Swarm Optimization (PSO), Density-Based Spatial Clustering (DBSCAN), and Greedy Search methods.
Key Results and Insights
The hybrid GNN-Transformer approach demonstrated significant performance gains over traditional methods, particularly as environmental complexity scaled.
- Superior Task Execution: In large-scale scenarios (e.g., 15 agents, 76 goals), the model achieved a 90% goal collection rate, a 110% relative improvement over the best-performing baseline DQN (42%).
- Complete Discovery: The model consistently achieved 100% grid coverage across all configurations, whereas baseline methods failed to explore larger maps fully.
- Operational Efficiency: The proposed method required an average of only 200 steps per episode, compared to the 600 steps required by PSO and standard RL approaches.
Analysis of the Transformer’s attention mechanism revealed it acts as a dynamic prioritization tool. Heatmaps show that agents learn to assign high weights (up to 1.0) to high-value goals while ignoring irrelevant nodes. Furthermore, ablation studies indicated a communication "sweet spot" around $k=5$ neighbors; beyond this, the system hit diminishing returns, suggesting that intelligent filtering of data is more critical than sheer volume.
Discussion and Future Work
While the current model excels in 2D grid environments, several avenues for future research remain. The current discrete action space does not yet account for the continuous 3D movement and complex aerodynamics of real-world drone flight. Additionally, the study assumes perfect communication within range; future iterations could incorporate "lossy" communication models. Finally, exploring heterogeneous swarms—where drones possess different sensor suites or battery capacities—would further enhance the system's real-world applicability.
Summary updated