Learned Multimodal Compression for Autonomous Driving

Hadi Hadizadeh, Ivan V. Bajic

TL;DR

This paper introduces a learned multimodal compression system designed specifically for autonomous driving tasks, such as 3D object detection. By compressing intermediate neural features rather than raw sensor data, the researchers achieved a staggering 2500:1 compression ratio on the nuScenes dataset while maintaining high detection accuracy, proving that "jointly coding" fused sensor data is significantly more efficient than compressing sensors individually.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) are data-hungry. A typical sensor suite—comprising multiple cameras, LiDAR, and radar—can generate upwards of 40 Gigabits per second (Gbps). Processing this volume of data locally consumes massive amounts of energy, while offloading it to the cloud for remote processing requires bandwidth that current wireless networks simply cannot provide.

Traditional compression (like JPEG or H.264) is designed for human eyes, focusing on visual fidelity. However, AV data is meant for machines. The authors address the "coding for machines" challenge: how do we strip away information that is irrelevant to the specific task (e.g., 3D object detection) while exploiting the massive redundancy between different sensors (e.g., a camera and LiDAR seeing the same car)?

Methods

The researchers built their system on top of FUTR3D, a state-of-the-art transformer-based sensor fusion framework. Instead of compressing raw pixels or point clouds, they extract intermediate "feature maps" from the network's backbones. These features have already passed through several layers, meaning much of the task-irrelevant noise has already been filtered out.

They explored three distinct coding architectures:

  1. Approach 1 (Joint Coding): The camera and LiDAR features are fused first into a single representation ($Z$), which is then compressed using a learned image codec called ANFIC (Augmented Normalizing Flows).
  2. Approach 2 (Conditional Coding - Camera First): Camera features are compressed first. Then, LiDAR features are compressed using CANF (Conditional ANF), which uses the already-decoded camera data as a "hint" to reduce redundancy.
  3. Approach 3 (Conditional Coding - LiDAR First): The reverse of Approach 2.

The core innovation lies in applying Normalizing Flows—a class of generative models—to map complex sensor data distributions into simpler, easier-to-compress forms.

Datasets

The study utilized the nuScenes dataset, a massive public benchmark for autonomous driving. It contains:

  • Approximately 34,000 data samples.
  • Data from 6 cameras, 5 radars, and 1 LiDAR sensor.
  • High-quality 3D bounding-box annotations for training and evaluation.

Key Insights and Ideas

  • The Power of Fusion: Approach 1 (joint coding of fused features) was the clear winner. The authors argue this is because the fusion process itself is "non-invertible"—it naturally discards information that isn't useful for the final 3D detection task before the compression even starts.
  • Information Bottleneck: The paper leans heavily on the Information Bottleneck principle. By moving deeper into the neural network before compressing, the system effectively "pre-filters" the data, ensuring the limited bit-budget is spent only on features that help the car "see" objects.
  • Asymmetric Predictability: They found that camera features are better at predicting LiDAR features than vice versa. This is likely because camera features are "smoother" and more similar to the natural images that the CANF codec was originally designed to handle.

Key Results

  • Extreme Compression: Approach 1 achieved the "no-compression" accuracy level at just 21 KBytes per sample. Given that the raw data for one sample is roughly 52 MBytes, this represents a compression ratio of nearly 2500:1.
  • Superiority over Standards: Approach 1 required ~30% lower bitrates than standard video coding (x265) while actually delivering higher detection accuracy.
  • Efficiency: In a head-to-head comparison (BD-Rate), Approach 1 was 67.7% more efficient than the worst-performing conditional coding method.
  • Latency: Approach 1 is the fastest of the three, with an on-board execution time of 0.67 seconds per sample on an NVIDIA RTX 2080 Ti, primarily because it only requires running a single codec.

Limitations and Future Work

While the results are promising, the authors acknowledge several areas for growth:

  • Modality Scope: The current study focused on Camera and LiDAR. Future work could integrate Radar data to further exploit inter-modal redundancies.
  • Real-Time Constraints: While 0.67 seconds per sample is a good start, it is still relatively slow for high-speed autonomous driving scenarios which require millisecond-level latency. Further optimization of the ANFIC/CANF architectures is needed.
  • Architecture Specificity: The findings are tied to the FUTR3D architecture; it remains to be seen if these insights hold true across different fusion models like BEVFormer or CenterPoint.

Summary updated

Abstract

Autonomous driving sensors generate an enormous amount of data. In this paper, we explore learned multimodal compression for autonomous driving, specifically targeted at 3D object detection. We focus on camera and LiDAR modalities and explore several coding approaches. One approach involves joint coding of fused modalities, while others involve coding one modality first, followed by conditional coding of the other modality. We evaluate the performance of these coding schemes on the nuScenes dataset. Our experimental results indicate that joint coding of fused modalities yields better results compared to the alternatives.
BibTeX
@inproceedings{hadizadeh2024learned,
  author = {Hadi Hadizadeh and Ivan V. Bajic},
  title = {{Learned Multimodal Compression for Autonomous Driving}},
  booktitle = {MMSP},
  year = {2024},
  url = {https://openreview.net/forum?id=kH8utlHlEA},
}

Loading related content...

Figure