MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation

Rui Liu, Zikang Wang, Peng Gao, Yu Shen, Pratap Tokekar, Ming C. Lin

Abstract

Autonomous vehicles (AVs) face significant challenges in complex urban environments where occlusions and sensor limitations can lead to accidents. While multi-modal sensing and connected autonomy offer potential solutions, they often rely on expensive hardware or perfect sensor availability. This paper presents MMCD (Multi-Modal Collaborative Decision-making), a framework that leverages cross-modal knowledge distillation to enhance the safety of connected AVs. By training a camera-based student model to mimic a LiDAR-enhanced teacher model, MMCD achieves high-performance decision-making even when high-end sensors are unavailable during deployment. Experimental results in CARLA and AirSim demonstrate an improvement in accident detection rates by up to 20.7% compared to existing benchmarks.

1. Introduction

The advancement of autonomous driving technology has reached a critical juncture where individual vehicle perception is often insufficient for safety-critical scenarios. Blind spots, occlusions from large vehicles, and unpredictable traffic violations at intersections remain "nightmare" scenarios for standalone AVs. Connected autonomy—where vehicles share data via Vehicle-to-Everything (V2X) communication—provides a broader perspective, yet it introduces new challenges regarding data heterogeneity and sensor reliability.

Current collaborative models typically assume the constant availability of high-fidelity sensors like LiDAR. However, LiDAR systems are costly and prone to failure or environmental interference. There is a pressing need for a system that maintains the spatial awareness of LiDAR while operating on more accessible and resilient modalities, such as RGB cameras.

2. The MMCD Framework

The MMCD framework addresses the gap between high-cost sensing and real-world deployment through a Teacher-Student architecture utilizing Knowledge Distillation (KD).

2.1 Cross-Modal Knowledge Distillation

The core innovation of MMCD lies in its training regime. A "Teacher" model is trained with access to the full suite of sensors: RGB camera images and LiDAR point clouds from both the ego vehicle and its collaborators. This model establishes a high-performance baseline for spatial reasoning.

A "Student" model is then trained to perform the same decision-making tasks using only RGB images. Through KD, the student model is penalized not just for incorrect decisions, but for deviating from the internal feature representations of the teacher. This process allows the student to learn "LiDAR-like" spatial features—such as depth perception and 3D object localization—from 2D image data.

2.2 Collaborative Fusion and Attention Mechanisms

To effectively process data from multiple sources, MMCD employs a sophisticated fusion strategy:

  • RGB Encoder: A ResNet-18 backbone extracts visual features, enhanced by a self-attention mechanism that prioritizes critical visual cues like traffic signals and pedestrians.
  • RGB Aggregator: Instead of simple data concatenation, which can be noisy, MMCD uses a cross-attention mechanism. This allows the ego vehicle to dynamically weigh contributions from collaborators based on their relevance to the current hazard.
  • Decision Logic: The fused features are processed by a Multi-Layer Perceptron (MLP) to output binary decisions (e.g., brake or proceed).

3. Experimental Evaluation

The researchers evaluated MMCD across two high-fidelity simulation platforms to test ground-to-ground and aerial-to-ground collaboration.

3.1 Simulation Environments

  • CARLA (with AutoCast): This environment was used to simulate three high-risk urban scenarios: risky overtaking on two-lane roads, obstructed left turns, and red-light violations.
  • AirSim: This platform facilitated the testing of aerial-ground collaboration, where a ground vehicle receives supplementary data from an overhead drone to navigate around large obstacles.

3.2 Performance Metrics

The primary metric for evaluation was the Accident Detection Rate (ADR). The study compared MMCD against the State-of-the-Art (SOTA) STG model.

4. Results and Discussion

The experimental results highlight the robustness and efficiency of the MMCD framework.

4.1 Safety and Robustness

MMCD demonstrated a significant safety boost, improving the ADR by up to 20.7% over the STG baseline. Crucially, in "Case D" testing—where the model was trained on multi-modal data but tested solely on RGB—the student model consistently outperformed standard camera-only models. This confirms that the knowledge distilled from LiDAR significantly enhances the camera's ability to interpret complex spatial scenes.

4.2 Aerial-Ground Synergy

In the AirSim experiments, the integration of drone-based perspectives led to a 7.0% improvement in accident detection. This suggests that heterogeneous collaborative networks (drones and cars) can effectively mitigate the limitations of ground-level occlusions.

4.3 Communication Efficiency

Practical V2X deployment requires low latency and minimal bandwidth usage. MMCD maintains a small communication footprint, with the student model requiring only 1.0KB for data exchange. This makes it highly suitable for current wireless communication standards.

5. Conclusion and Future Work

MMCD represents a significant step toward making connected autonomy both safer and more practical. By leveraging knowledge distillation, the framework allows vehicles to benefit from high-end sensor logic without the associated hardware costs or reliability issues.

Future research will focus on addressing current limitations, including the impact of GPS noise on collaborative positioning and the integration of additional modalities like radar or infrared sensors to improve performance in adverse weather conditions. Furthermore, incorporating state estimation techniques will be vital to managing the latency and noise inherent in real-world V2V networks.

Summary updated

Abstract

Autonomous systems have advanced significantly, but challenges persist in accident-prone environments where robust decision-making is crucial. A single vehicle's limited sensor range and obstructed views increase the likelihood of accidents. Multi-vehicle connected systems and multi-modal approaches, leveraging RGB images and LiDAR point clouds, have emerged as promising solutions. However, existing methods often assume the availability of all data modalities and connected vehicles during both training and testing, which is impractical due to potential sensor failures or missing connected vehicles. To address these challenges, we introduce a novel framework MMCD (Multi-Modal Collaborative Decision-making) for connected autonomy. Our framework fuses multi-modal observations from ego and collaborative vehicles to enhance decision-making under challenging conditions. To ensure robust performance when certain data modalities are unavailable during testing, we propose an approach based on cross-modal knowledge distillation with a teacher-student model structure. The teacher model is trained with multiple data modalities, while the student model is designed to operate effectively with reduced modalities. In experiments on $\textit{connected autonomous driving with ground vehicles}$ and $\textit{aerial-ground vehicles collaboration}$, our method improves driving safety by up to ${\it 20.7}\%$, surpassing the best-existing baseline in detecting potential accidents and making safe driving decisions. More information can be found on our website https://ruiiu.github.io/mmcd.
BibTeX
@inproceedings{liu2025mmcd,
  author = {Rui Liu and Zikang Wang and Peng Gao and Yu Shen and Pratap Tokekar and Ming C. Lin},
  title = {{MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246798},
}
Figure