MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models

Sourav Biswas, Jerry Liu, Kelvin Wong, Shenlong Wang, Raquel Urtasun

TL;DR

MuSCLE is a novel compression framework that slashes the storage requirements of LiDAR data streams by exploiting the "memory" of the sensor. By using deep entropy models to capture spatio-temporal redundancies across successive LiDAR sweeps, it achieves a 7–35% bitrate reduction over previous state-of-the-art methods while maintaining high accuracy for downstream tasks like object detection and semantic segmentation.

What Problem Does This Paper Address?

LiDAR sensors are the "eyes" of autonomous vehicles, but they are incredibly chatty. A single 64-line sensor can generate over 3 billion points per hour. Storing and transmitting this massive influx of 3D data is a logistical nightmare for engineers.

While image and video compression are mature fields, point cloud compression is much harder because the data is sparse, unstructured, and 3D. Most existing solutions treat each LiDAR "sweep" (a 360-degree rotation) as an independent event, ignoring the fact that the world doesn't change much in the 100 milliseconds between frames. Furthermore, most methods focus only on the geometry (the $x, y, z$ coordinates) and neglect the "intensity" (the reflectivity of the surface), which is vital for many AI perception models.

Methods

The authors propose MuSCLE (Multi Sweep Compression of LiDAR using Entropy models) to address these inefficiencies. The pipeline works in several key stages to jointly compress geometry and intensity:

MuSCLE Architecture Pipeline
  1. Octree Representation: The 3D space is partitioned into an octree—a tree structure where each node represents a cube of space. If a cube contains points, it's subdivided into eight smaller "octants." This continues until a desired precision is reached.
  2. Conditional Entropy Modeling: This is the core innovation. Instead of using a static probability model to encode the octree, MuSCLE uses a deep neural network to predict the probability of a node being occupied based on two contexts:
  3. Ancestral context: Information from higher levels of the current tree (top-down pass).
  4. Temporal context: Information from the previous LiDAR sweep.
  5. Continuous Convolutions: Because points in two different sweeps never align perfectly, the model uses "continuous convolutions" to aggregate features from the previous frame. This allows the model to "look" at the neighborhood of a point in the past to predict its state in the present without requiring strict voxel grids.
  6. Intensity Compression: The same spatio-temporal logic is applied to intensity values. By predicting the intensity of a point based on its neighbors in the previous frame, the model can encode the actual values using far fewer bits.
  7. Lossless Coding: The predicted probabilities are fed into an arithmetic range coder to produce the final, highly compressed bitstream.

Datasets

The researchers evaluated MuSCLE on two massive, industry-standard datasets:

  • UrbanCity: A large-scale dataset collected by self-driving vehicles across North American cities, featuring 5,000 sequences.
  • SemanticKITTI: A popular benchmark for LiDAR point cloud sequences that includes dense semantic labels for every point.

Key Insights and Ideas

  • Temporal Redundancy is King: By simply looking at the previous frame, the researchers reduced the spatial bitrate by up to 45% on SemanticKITTI. This confirms that the "static" parts of the world (roads, buildings) provide a massive opportunity for compression.
  • Continuous Convolutions as an Aligner: Traditional convolutions require a grid. By using continuous convolutions, MuSCLE can relate unstructured 3D points across time without needing explicit "tracking" or point-to-point correspondences.
  • Preserving "Realism": The paper argues that bitrate isn't the only metric that matters. They show that AI models (like segmentors) perform better on MuSCLE-compressed data than on data compressed by other methods at the same bitrate. This suggests MuSCLE preserves the features that actually matter for machine "vision."

Key Results

MuSCLE demonstrates state-of-the-art performance across multiple metrics, proving highly effective at reducing bitrates while preserving geometric fidelity.

Rate-Distortion Performance on UrbanCity and KITTI
  • Bitrate Reduction: MuSCLE achieved a 7–17% reduction on UrbanCity and a 15–35% reduction (joint geometry and intensity) on SemanticKITTI compared to OctSqueeze, the previous state-of-the-art deep learning method.
  • Superiority over Standards: It significantly outperformed the MPEG TMC13 standard, especially at lower bitrates where MPEG tends to produce more artifacts. This is evident in the rate-distortion curves where MuSCLE maintains higher F1 scores and PSNR at lower bits-per-point.
  • Downstream Performance: When testing on object detection (AP) and semantic segmentation (mIoU), MuSCLE-compressed data showed the smallest "realism gap," meaning the AI's performance stayed closest to what it would be on uncompressed "Oracle" data.

Limitations and Future Work

While MuSCLE is a significant step forward, the authors note a few areas for growth:

  • Sensor Suites: Currently, the model only handles LiDAR. The authors suggest that future work could involve "cross-modal" compression—using camera images to help compress LiDAR data or vice versa.
  • Computational Complexity: The use of continuous convolutions and deep entropy models is computationally more intensive than simple geometric codecs. While the paper mentions parallelization, real-time deployment on edge hardware remains a challenge for deep learning-based compression.
  • Sequential Dependency: Because the model relies on the previous sweep, a single corrupted frame could theoretically impact the decoding of subsequent frames (a common issue in video compression known as error propagation).

Summary updated

Abstract

We present a novel compression algorithm for reducing the storage of LiDAR sensory data streams. Our model exploits spatio-temporal relationships across multiple LIDAR sweeps to reduce the bitrate of both geometry and intensity values. Towards this goal, we propose a novel conditional entropy model that models the probabilities of the octree symbols, by considering both coarse level geometry and previous sweeps’ geometric and intensity information. We then exploit the learned probability to encode the full data-stream into a compact one. Our experiments demonstrate that our method significantly reduces the joint geometry and intensity bitrate over prior state-of-the-art LiDAR compression methods, with a reduction of 7–17% and 15–35% on the UrbanCity and SemanticKITTI datasets respectively.
BibTeX
@inproceedings{biswas2020muscle,
  author = {Sourav Biswas and Jerry Liu and Kelvin Wong and Shenlong Wang and Raquel Urtasun},
  title = {{MuSCLE: Multi Sweep Compression of LiDAR using Deep Entropy Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=YlvtABtiKX},
}
Figure