MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation

Rui Liu, Zikang Wang, Peng Gao, Yu Shen, Pratap Tokekar, Ming Lin

Abstract

Autonomous vehicles (AVs) face significant challenges in complex urban environments where occlusions and sensor limitations can lead to accidents. While multi-modal sensing and connected autonomy offer potential solutions, they often rely on expensive hardware or perfect sensor availability. This paper presents MMCD (Multi-Modal Collaborative Decision-making), a framework that leverages cross-modal knowledge distillation to enhance the safety of connected AVs. By training a camera-based student model to mimic a LiDAR-enhanced teacher model, MMCD achieves high-performance decision-making even when high-end sensors are unavailable during deployment. Experimental results in CARLA and AirSim demonstrate an improvement in accident detection rates by up to 20.7% compared to existing benchmarks.

1. Introduction

The advancement of autonomous driving technology has reached a critical juncture where individual vehicle perception is often insufficient for safety-critical scenarios. Blind spots, occlusions from large vehicles, and unpredictable traffic violations at intersections remain "nightmare" scenarios for standalone AVs. Connected autonomy—where vehicles share data via Vehicle-to-Everything (V2X) communication—provides a broader perspective, yet it introduces new challenges regarding data heterogeneity and sensor reliability.

Current collaborative models typically assume the constant availability of high-fidelity sensors like LiDAR. However, LiDAR systems are costly and prone to failure or environmental interference. There is a pressing need for a system that maintains the spatial awareness of LiDAR while operating on more accessible and resilient modalities, such as RGB cameras.

2. The MMCD Framework

The MMCD framework addresses the gap between high-cost sensing and real-world deployment through a Teacher-Student architecture utilizing Knowledge Distillation (KD).

2.1 Cross-Modal Knowledge Distillation

The core innovation of MMCD lies in its training regime. A "Teacher" model is trained with access to the full suite of sensors: RGB camera images and LiDAR point clouds from both the ego vehicle and its collaborators. This model establishes a high-performance baseline for spatial reasoning.

A "Student" model is then trained to perform the same decision-making tasks using only RGB images. Through KD, the student model is penalized not just for incorrect decisions, but for deviating from the internal feature representations of the teacher. This process allows the student to learn "LiDAR-like" spatial features—such as depth perception and 3D object localization—from 2D image data.

2.2 Collaborative Fusion and Attention Mechanisms

To effectively process data from multiple sources, MMCD employs a sophisticated fusion strategy:

  • RGB Encoder: A ResNet-18 backbone extracts visual features, enhanced by a self-attention mechanism that prioritizes critical visual cues like traffic signals and pedestrians.
  • RGB Aggregator: Instead of simple data concatenation, which can be noisy, MMCD uses a cross-attention mechanism. This allows the ego vehicle to dynamically weigh contributions from collaborators based on their relevance to the current hazard.
  • Decision Logic: The fused features are processed by a Multi-Layer Perceptron (MLP) to output binary decisions (e.g., brake or proceed).

3. Experimental Evaluation

The researchers evaluated MMCD across two high-fidelity simulation platforms to test ground-to-ground and aerial-to-ground collaboration.

3.1 Simulation Environments

  • CARLA (with AutoCast): This environment was used to simulate three high-risk urban scenarios: risky overtaking on two-lane roads, obstructed left turns, and red-light violations.
  • AirSim: This platform facilitated the testing of aerial-ground collaboration, where a ground vehicle receives supplementary data from an overhead drone to navigate around large obstacles.

3.2 Performance Metrics

The primary metric for evaluation was the Accident Detection Rate (ADR). The study compared MMCD against the State-of-the-Art (SOTA) STG model.

4. Results and Discussion

The experimental results highlight the robustness and efficiency of the MMCD framework.

4.1 Safety and Robustness

MMCD demonstrated a significant safety boost, improving the ADR by up to 20.7% over the STG baseline. Crucially, in "Case D" testing—where the model was trained on multi-modal data but tested solely on RGB—the student model consistently outperformed standard camera-only models. This confirms that the knowledge distilled from LiDAR significantly enhances the camera's ability to interpret complex spatial scenes.

4.2 Aerial-Ground Synergy

In the AirSim experiments, the integration of drone-based perspectives led to a 7.0% improvement in accident detection. This suggests that heterogeneous collaborative networks (drones and cars) can effectively mitigate the limitations of ground-level occlusions.

4.3 Communication Efficiency

Practical V2X deployment requires low latency and minimal bandwidth usage. MMCD maintains a small communication footprint, with the student model requiring only 1.0KB for data exchange. This makes it highly suitable for current wireless communication standards.

5. Conclusion and Future Work

MMCD represents a significant step toward making connected autonomy both safer and more practical. By leveraging knowledge distillation, the framework allows vehicles to benefit from high-end sensor logic without the associated hardware costs or reliability issues.

Future research will focus on addressing current limitations, including the impact of GPS noise on collaborative positioning and the integration of additional modalities like radar or infrared sensors to improve performance in adverse weather conditions. Furthermore, incorporating state estimation techniques will be vital to managing the latency and noise inherent in real-world V2V networks.

Summary updated

BibTeX
@inproceedings{liu2025mmcd,
  author = {Rui Liu and Zikang Wang and Peng Gao and Yu Shen and Pratap Tokekar and Ming Lin},
  title = {{MMCD: Multi-Modal Collaborative Decision-Making for Connected Autonomy with Knowledge Distillation}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2025},
  doi = {10.1109/iros60139.2025.11246798},
}
Figure