Abstract
Autonomous vehicles (AVs) face significant challenges in complex urban environments where occlusions and sensor limitations can lead to accidents. While multi-modal sensing and connected autonomy offer potential solutions, they often rely on expensive hardware or perfect sensor availability. This paper presents MMCD (Multi-Modal Collaborative Decision-making), a framework that leverages cross-modal knowledge distillation to enhance the safety of connected AVs. By training a camera-based student model to mimic a LiDAR-enhanced teacher model, MMCD achieves high-performance decision-making even when high-end sensors are unavailable during deployment. Experimental results in CARLA and AirSim demonstrate an improvement in accident detection rates by up to 20.7% compared to existing benchmarks.
1. Introduction
The advancement of autonomous driving technology has reached a critical juncture where individual vehicle perception is often insufficient for safety-critical scenarios. Blind spots, occlusions from large vehicles, and unpredictable traffic violations at intersections remain "nightmare" scenarios for standalone AVs. Connected autonomy—where vehicles share data via Vehicle-to-Everything (V2X) communication—provides a broader perspective, yet it introduces new challenges regarding data heterogeneity and sensor reliability.
Current collaborative models typically assume the constant availability of high-fidelity sensors like LiDAR. However, LiDAR systems are costly and prone to failure or environmental interference. There is a pressing need for a system that maintains the spatial awareness of LiDAR while operating on more accessible and resilient modalities, such as RGB cameras.
2. The MMCD Framework
The MMCD framework addresses the gap between high-cost sensing and real-world deployment through a Teacher-Student architecture utilizing Knowledge Distillation (KD).
2.1 Cross-Modal Knowledge Distillation
The core innovation of MMCD lies in its training regime. A "Teacher" model is trained with access to the full suite of sensors: RGB camera images and LiDAR point clouds from both the ego vehicle and its collaborators. This model establishes a high-performance baseline for spatial reasoning.
A "Student" model is then trained to perform the same decision-making tasks using only RGB images. Through KD, the student model is penalized not just for incorrect decisions, but for deviating from the internal feature representations of the teacher. This process allows the student to learn "LiDAR-like" spatial features—such as depth perception and 3D object localization—from 2D image data.
2.2 Collaborative Fusion and Attention Mechanisms
To effectively process data from multiple sources, MMCD employs a sophisticated fusion strategy:
- RGB Encoder: A ResNet-18 backbone extracts visual features, enhanced by a self-attention mechanism that prioritizes critical visual cues like traffic signals and pedestrians.
- RGB Aggregator: Instead of simple data concatenation, which can be noisy, MMCD uses a cross-attention mechanism. This allows the ego vehicle to dynamically weigh contributions from collaborators based on their relevance to the current hazard.
- Decision Logic: The fused features are processed by a Multi-Layer Perceptron (MLP) to output binary decisions (e.g., brake or proceed).
3. Experimental Evaluation
The researchers evaluated MMCD across two high-fidelity simulation platforms to test ground-to-ground and aerial-to-ground collaboration.
3.1 Simulation Environments
- CARLA (with AutoCast): This environment was used to simulate three high-risk urban scenarios: risky overtaking on two-lane roads, obstructed left turns, and red-light violations.
- AirSim: This platform facilitated the testing of aerial-ground collaboration, where a ground vehicle receives supplementary data from an overhead drone to navigate around large obstacles.
3.2 Performance Metrics
The primary metric for evaluation was the Accident Detection Rate (ADR). The study compared MMCD against the State-of-the-Art (SOTA) STG model.
4. Results and Discussion
The experimental results highlight the robustness and efficiency of the MMCD framework.
4.1 Safety and Robustness
MMCD demonstrated a significant safety boost, improving the ADR by up to 20.7% over the STG baseline. Crucially, in "Case D" testing—where the model was trained on multi-modal data but tested solely on RGB—the student model consistently outperformed standard camera-only models. This confirms that the knowledge distilled from LiDAR significantly enhances the camera's ability to interpret complex spatial scenes.
4.2 Aerial-Ground Synergy
In the AirSim experiments, the integration of drone-based perspectives led to a 7.0% improvement in accident detection. This suggests that heterogeneous collaborative networks (drones and cars) can effectively mitigate the limitations of ground-level occlusions.
4.3 Communication Efficiency
Practical V2X deployment requires low latency and minimal bandwidth usage. MMCD maintains a small communication footprint, with the student model requiring only 1.0KB for data exchange. This makes it highly suitable for current wireless communication standards.
5. Conclusion and Future Work
MMCD represents a significant step toward making connected autonomy both safer and more practical. By leveraging knowledge distillation, the framework allows vehicles to benefit from high-end sensor logic without the associated hardware costs or reliability issues.
Future research will focus on addressing current limitations, including the impact of GPS noise on collaborative positioning and the integration of additional modalities like radar or infrared sensors to improve performance in adverse weather conditions. Furthermore, incorporating state estimation techniques will be vital to managing the latency and noise inherent in real-world V2V networks.
Summary updated