Abstract
Traditional Reinforcement Learning (RL) suffers from replicating human-like behaviors, generalizing effectively in multi-agent scenarios, and overcoming inherent interpretability issues. These tasks are compounded when deep environment understanding, agent coordination and dynamic optimization are required. While Large Language Model (LLM) enhanced methods have shown promise in generalization and interoperability, they often neglect necessary multi-agent coordination. Therefore, we introduce the Cascading Cooperative Multi-agent (CCMA) framework, integrating RL for individual interactions, a fine-tuned LLM for regional cooperation, a reward function for global optimization, and the Retrieval-augmented Generation mechanism to dynamically optimize decision-making across complex driving scenarios. Our experiments demonstrate that the CCMA outperforms existing RL methods, demonstrating significant improvements in both micro and macro-level performance in complex driving environments.
Introduction
Highway on-ramp merging represents a significant challenge for autonomous driving systems, requiring a delicate balance between individual safety and collective traffic efficiency. Traditional Multi-agent Reinforcement Learning (MARL) approaches often struggle with sample inefficiency and a lack of interpretability, frequently failing to generalize across diverse road layouts. While Large Language Models (LLMs) offer advanced reasoning capabilities, they are typically applied to single-agent scenarios, overlooking the regional coordination necessary for complex maneuvers like merging.
To bridge this gap, we propose the Cascading Cooperative Multi-agent (CCMA) framework. This system combines the high-speed "reflexes" of RL with the high-level "reasoning" of LLMs, organized into a three-tier hierarchy—individual, regional, and global—to achieve superior coordination and performance in dense traffic environments.
The CCMA Framework
The CCMA framework decomposes the complex task of multi-agent coordination into three cascading levels of decision-making:
- Individual Level (The Reflex): At the lowest level, pre-trained RL models handle immediate, high-responsiveness tasks. This includes basic lane keeping and emergency hazard avoidance, ensuring the vehicle maintains fundamental safety without the latency of language processing.
- Regional Level (The Coordination): This level utilizes a fine-tuned Visual Language Model (VLM), specifically GLM-4v-9B, to manage spatial reasoning and multi-vehicle intent. By processing both numerical state data and visual screenshots from the simulator, the VLM predicts vehicle intents and facilitates a structured language-based coordination protocol.
- Global Level (The Strategy): The highest level employs a Retrieval-Augmented Generation (RAG) strategy to optimize the system's overarching goals. An LLM dynamically redesigns reward functions based on evolving traffic conditions, storing successful strategies in a database for future retrieval when similar patterns occur.
The integration of these levels allows the system to remain interpretable; humans can inspect the "intent" vectors generated at the regional level to understand the cooperative logic behind specific maneuvers.
Experimental Setup and Results
The framework was evaluated using a modified highway-env simulation environment, featuring approximately 50,000 samples across six different road maps. The simulations tested three traffic densities: Easy, Medium, and Hard.
The CCMA framework consistently outperformed state-of-the-art MARL benchmarks, including MAPPO, MAA2C, and MAACKTR. As shown in the performance curves, the CCMA method (labeled "Ours") maintains a higher success rate and faster convergence across all difficulty levels.
In high-density "Hard" scenarios, CCMA achieved a success rate of 0.72, a significant improvement over traditional methods that often fail to coordinate in tight spatial constraints. In "Easy" mode, the success rate reached 0.89. In low-density conditions, the system achieved an average merging time of 21 seconds with zero accidents. Furthermore, ablation studies confirmed that the inclusion of the RAG mechanism and regional LLM coordination provided a 10-15% performance boost over using RL alone.
Conclusion and Future Work
The CCMA framework demonstrates that the synergy between RL and LLMs can effectively solve complex multi-agent coordination problems. By leveraging RL for low-level control and LLMs for high-level social reasoning and strategic optimization, the system achieves high performance without sacrificing interpretability.
Future research will focus on addressing the inference latency of large models in real-time physical environments and expanding the framework's application to even more chaotic scenarios, such as unprotected urban intersections and environments with high pedestrian density.
Summary updated