Scalable Human-Machine Point Cloud Compression

Mateen Ulhaq, Ivan V. Bajic

TL;DR

Researchers have developed the first "scalable" point cloud codec that prioritizes machine classification at ultra-low bitrates while allowing for high-quality human reconstruction. By splitting the compressed data into a "base" layer for AI and an "enhancement" layer for humans, the system achieves superior accuracy-per-bit compared to traditional compression methods.

What Problem Does This Paper Address?

As 3D point clouds become central to robotics, autonomous driving, and AR, we face a "compute vs. bandwidth" bottleneck. Edge devices (like drones or sensors) often lack the power to run complex deep learning models locally. The standard solution is to compress the data and send it to a server.

However, traditional codecs (like Draco or G-PCC) are designed for human viewing—they spend bits on visual details that a machine learning model might not need to identify an object. Conversely, previous "coding for machines" research focused solely on AI tasks, leaving humans in the dark if they needed to verify the data. This paper addresses the need for a multi-task codec that can efficiently serve both machines and humans without redundant data transmission.

Methods

The authors propose a scalable architecture based on PointNet++, a hierarchical deep learning model for point clouds.

  1. Hierarchical Downsampling: The encoder uses a series of "set abstraction" layers to progressively reduce the number of points while extracting high-level features.
  2. Latent Space Splitting: At the bottleneck of the network, the compressed representation (latent space) is split into two parts:
  3. Base Layer ($y_1$): A highly compact stream containing only the information necessary for the machine task (classification).
  4. Enhancement Layer ($y_2$): Additional data that, when combined with the base layer, allows a decoder to reconstruct the original 3D shape for human review.
  5. The "Detach" Trick: During training, the authors used a detach() operation. This prevents the gradients from the human reconstruction task from "polluting" the base layer. This ensures the base layer remains strictly specialized for classification, maximizing its efficiency.
  6. Entropy Coding: The latent representations are quantized and losslessly compressed using learned entropy models to further reduce file size.

Datasets

The model was trained and evaluated on ModelNet40, a standard benchmark dataset containing 12,311 CAD models from 40 object categories (e.g., airplanes, chairs, lamps). The points were sampled and scaled to 1024 points per object for the experiments.

Key Insights and Ideas

  • Task Specialization: Machines are "lazy" in a good way—they don't need a perfect 3D mesh to know they are looking at a car. By stripping away task-irrelevant information, the base bitstream can be incredibly small.
  • Scalability is Efficiency: Instead of sending two different files for two different purposes, the enhancement layer simply "adds" to the base layer. This avoids the redundancy of sending the same core features twice.
  • PointNet++ Superiority: By moving from the original PointNet (used in previous work) to PointNet++, the authors leveraged local geometric structures, leading to better classification accuracy at lower bitrates.

Key Results

  • Classification Performance: The proposed codec significantly outperformed existing specialized codecs (like "Ulhaq 2023") and non-specialized "Input Compression Codecs" (ICC). It achieved higher Top-1 accuracy at much lower bitrates (measured in bits per point, or bpp).
  • The Gap: There is a massive performance gap between this task-specialized approach and standard codecs like TMC13 or OctAttention when used for machine tasks. The specialized codec removes "statistical redundancy" and "task-irrelevant information" that standard codecs keep.
  • Reconstruction Quality: For human viewing, the proposed codec achieved better reconstruction quality (measured by Chamfer distance) than conventional codecs at low bitrates (below 1.6 bpp). While standard codecs eventually catch up at high bitrates, the scalable codec wins in bandwidth-constrained scenarios.

Limitations and Future Work

  • Task Variety: This paper focused specifically on classification. Future work is needed to see how this scalable approach handles more complex tasks like semantic segmentation (labeling every point in a scene) or object detection.
  • Dataset Complexity: ModelNet40 consists of isolated objects. Real-world applications involve complex, "noisy" outdoor scenes (like a street view from a LiDAR sensor), which will be a much tougher test for the codec.
  • Architectural Refinement: The authors used "Single-Scale Grouping" for simplicity; they suggest that "Multi-Scale Grouping" (MSG) could further improve performance by capturing features at different densities.

Summary updated

Abstract

Due to the limited computational capabilities of edge devices, deep learning inference can be quite expensive. One remedy is to compress and transmit point cloud data over the network for server-side processing. Unfortunately, this approach can be sensitive to network factors, including available bitrate. Luckily, the bitrate requirements can be reduced without sacrificing inference accuracy by using a machine task-specialized codec. In this paper, we present a scalable codec for point-cloud data that is specialized for the machine task of classification, while also providing a mechanism for human viewing. In the proposed scalable codec, the “base” bitstream supports the machine task, and an “enhancement” bitstream may be used for better input reconstruction performance for human viewing. We base our architecture on PointNet++, and test its efficacy on the ModelNet40 dataset. We show significant improvements over prior non-specialized codecs.
BibTeX
@inproceedings{ulhaq2024scalable,
  author = {Mateen Ulhaq and Ivan V. Bajic},
  title = {{Scalable Human-Machine Point Cloud Compression}},
  booktitle = {PCS},
  year = {2024},
  url = {https://openreview.net/forum?id=g6zbqrjwRy},
}
Figure