MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models

Sourav Biswas, Jerry Liu, Kelvin Wong, Shenlong Wang, Raquel Urtasun

Abstract

We present a novel compression algorithm for reducing the storage of LiDAR sensor data streams. Our model, MuSCLE (Multi Sweep Compression of LiDAR), exploits spatio-temporal relationships across multiple LiDAR sweeps to reduce the bitrate of both geometry and intensity values. Towards this goal, we propose a novel conditional entropy model that models the probabilities of the octree symbols by considering both coarse level geometry and previous sweeps' geometric and intensity information. We then use the learned probability to encode the full data stream into a compact one. Our experiments demonstrate that our method significantly reduces the joint geometry and intensity bitrate over prior state-of-the-art LiDAR compression methods, with a reduction of 7-17% and 15-35% on the UrbanCity and SemanticKITTI datasets respectively.

Introduction

LiDAR sensors are the primary "eyes" of autonomous vehicles, but they are incredibly data-hungry. A single 64-line sensor can generate over 3 billion points per hour, creating a massive bottleneck for data storage and wireless transmission. While image and video compression are mature fields, 3D point cloud compression is significantly more challenging because the data is sparse, unstructured, and lacks a fixed grid.

Most existing methods either compress each "sweep" (a 360-degree scan) independently—ignoring temporal redundancy—or focus solely on 3D coordinates (geometry) while neglecting reflection data (intensity). MuSCLE bridges this gap by creating a unified framework for spatio-temporal LiDAR stream compression, jointly modeling geometry and intensity to achieve state-of-the-art efficiency.

Methodology

MuSCLE represents the point cloud using an octree, a tree-based data structure that recursively partitions 3D space. The core of the system is a Deep Conditional Entropy Model that predicts the probability of octree symbols and intensity values. By accurately predicting the next piece of data based on context, the model minimizes the bits required for encoding.

Overview of the MuSCLE framework, illustrating the octree representation and the entropy models for occupancy and intensity.

Spatio-Temporal Context and Continuous Convolutions

Unlike prior models that only look at the current sweep, MuSCLE leverages the previous sweep to identify patterns. Since points in consecutive sweeps do not perfectly align due to sensor motion, the model uses continuous convolutions to aggregate features from neighboring points in the previous sweep, effectively "aligning" the temporal data.

Two-Stream Feature Backbone

The model employs a two-stream backbone for the previous sweep, running both a top-down pass (ancestral information) and a bottom-up pass (descendant information) to create a rich representation of the scene's history. This joint modeling recognizes that geometry and intensity are highly correlated; for instance, an object's shape often dictates its reflective properties.

Experiments

The model was evaluated on two major large-scale LiDAR benchmarks:

  1. UrbanCity: A dataset collected across North American cities, containing 5,000 training sequences and 500 test sequences.
  2. SemanticKITTI: A popular benchmark based on the KITTI vision suite, providing dense semantic labels which allowed testing how compression affects downstream AI perception tasks.

Results

MuSCLE achieved a 7–17% reduction in joint geometry and intensity bitrate on UrbanCity and a 15–35% reduction on SemanticKITTI compared to prior deep-learning leaders like OctSqueeze. It significantly outperformed the MPEG TMC13 standard, particularly at lower bitrates.

Rate-distortion curves comparing MuSCLE against state-of-the-art methods on UrbanCity and SemanticKITTI datasets across various metrics including F1 score, Chamfer Distance, and PSNR.

Beyond raw bitrate efficiency, MuSCLE reconstructions proved more "realistic" for downstream AI models. When compressed data was fed into object detection and segmentation models, MuSCLE maintained higher Mean IoU (Intersection over Union) and Average Precision (AP) than competing codecs, demonstrating that it preserves the features most critical for autonomous driving safety.

Conclusion

By exploiting temporal redundancy and joint geometry-intensity modeling, MuSCLE sets a new standard for LiDAR stream compression. Future work will explore entropy modeling for "leaf offsets" (high-frequency artifacts currently left uncompressed) and extending the model to longer temporal windows. Additionally, the authors aim to apply these principles to the joint compression of entire multi-modal sensor suites.

Summary updated

Abstract

We present a novel compression algorithm for reducing the storage of LiDAR sensor data streams. Our model exploits spatio-temporal relationships across multiple LiDAR sweeps to reduce the bitrate of both geometry and intensity values. Towards this goal, we propose a novel conditional entropy model that models the probabilities of the octree symbols by considering both coarse level geometry and previous sweeps' geometric and intensity information. We then use the learned probability to encode the full data stream into a compact one. Our experiments demonstrate that our method significantly reduces the joint geometry and intensity bitrate over prior state-of-the-art LiDAR compression methods, with a reduction of 7-17% and 15-35% on the UrbanCity and SemanticKITTI datasets respectively.
BibTeX
@inproceedings{biswas2020muscle,
  author = {Sourav Biswas and Jerry Liu and Kelvin Wong and Shenlong Wang and Raquel Urtasun},
  title = {{MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=YlvtABtiKX},
}
Figure