Learned Multimodal Compression for Autonomous Driving

Hadi Hadizadeh, Ivan V. Bajic

Abstract

Autonomous driving sensors generate an enormous amount of data. In this paper, we explore learned multimodal compression for autonomous driving, specifically targeted at 3D object detection. We focus on camera and LiDAR modalities and explore several coding approaches. One approach involves joint coding of fused modalities, while others involve coding one modality first, followed by conditional coding of the other modality. We evaluate the performance of these coding schemes on the nuScenes dataset. Our experimental results indicate that joint coding of fused modalities yields better results compared to the alternatives.

Introduction

Autonomous vehicles (AVs) are equipped with a suite of sensors—including multiple cameras, LiDAR, and radar—that can generate upwards of 40 Gigabits per second (Gbps). Processing this data locally is energy-intensive, while offloading it to the cloud for remote processing is constrained by the bandwidth of current wireless networks. Traditional compression standards like JPEG or H.264 are optimized for human perception, focusing on visual fidelity. However, AV data is primarily consumed by machines for tasks like 3D object detection. This paper addresses the "coding for machines" challenge by stripping away task-irrelevant information and exploiting redundancies between different sensor modalities.

Methodology

The proposed system is built upon FUTR3D, a transformer-based sensor fusion framework. Instead of compressing raw sensor data, the system extracts and compresses intermediate "feature maps" from the network's backbones. These features have already undergone initial processing, filtering out much of the task-irrelevant noise.

The researchers explored three distinct coding architectures:

  1. Approach 1 (Joint Coding): Camera and LiDAR features are fused into a single representation ($Z$), which is then compressed using a learned image codec called ANFIC (Augmented Normalizing Flows).
  2. Approach 2 (Conditional Coding - Camera First): Camera features are compressed first. LiDAR features are then compressed using CANF (Conditional ANF), using the decoded camera data as a side information "hint".
  3. Approach 3 (Conditional Coding - LiDAR First): Similar to Approach 2, but LiDAR is compressed first.
Figure 1: Architecture of Approach 1, where fused features are jointly compressed.

The core innovation is the application of Normalizing Flows to map complex sensor data distributions into simpler, more compressible forms.

Experimental Results

The system was evaluated using the nuScenes dataset, which includes data from 6 cameras, 5 radars, and 1 LiDAR sensor across 34,000 samples.

The experimental results demonstrate that Approach 1 (Joint Coding) is the most effective. It achieved the "no-compression" accuracy level at a bitrate of just 21 KBytes per sample. Given the raw data size of approximately 52 MBytes per sample, this represents a compression ratio of nearly 2500:1.

Figure 3: Rate-distortion curves comparing the three approaches against the no-compression baseline.

As shown in Figure 3, Approach 1 consistently outperforms the conditional coding methods. In terms of Bjøntegaard Delta Rate (BD-Rate), Approach 1 was found to be 67.7% more efficient than the worst-performing conditional method. Furthermore, Approach 1 required approximately 30% lower bitrates than standard video coding (x265) while maintaining higher detection accuracy.

Discussion and Conclusion

The superiority of joint coding (Approach 1) stems from the fusion process itself. Because fusion is "non-invertible," it naturally discards information that is not useful for the final 3D detection task before compression begins. This aligns with the Information Bottleneck principle, ensuring the bit-budget is spent only on task-relevant features.

Additionally, Approach 1 is the most computationally efficient, with an on-board execution time of 0.67 seconds per sample on an NVIDIA RTX 2080 Ti. While this is a significant step forward, future work is needed to achieve the millisecond-level latency required for real-time high-speed driving. Expanding the scope to include radar data and testing across different fusion architectures like BEVFormer are also key areas for future research.

Summary updated

Abstract

Autonomous driving sensors generate an enormous amount of data. In this paper, we explore learned multimodal compression for autonomous driving, specifically targeted at 3D object detection. We focus on camera and LiDAR modalities and explore several coding approaches. One approach involves joint coding of fused modalities, while others involve coding one modality first, followed by conditional coding of the other modality. We evaluate the performance of these coding schemes on the nuScenes dataset. Our experimental results indicate that joint coding of fused modalities yields better results compared to the alternatives.
BibTeX
@inproceedings{hadizadeh2024learned,
  author = {Hadi Hadizadeh and Ivan V. Bajic},
  title = {{Learned Multimodal Compression for Autonomous Driving}},
  booktitle = {MMSP},
  year = {2024},
  url = {https://openreview.net/forum?id=kH8utlHlEA},
}
Figure