ALSO: Automotive Lidar Self-supervision by Occupancy estimation

Alexandre Boulch, Corentin Sautier, Björn Michele, Gilles Puy, Renaud Marlet

TL;DR

ALSO (Automotive Lidar Self-supervision by Occupancy estimation) is a simple, memory-efficient method for pre-training 3D perception models without manual labels. By training a network to reconstruct scene surfaces through occupancy estimation—using only raw sensor visibility as a guide—it learns rich semantic features that significantly boost performance in downstream tasks like semantic segmentation and object detection.

What Problem Does This Paper Address?

Deep learning has revolutionized autonomous driving, but it has a massive "data hunger" problem. While collecting raw Lidar data is easy, annotating it is a nightmare: labeling 3D point clouds is orders of magnitude more expensive and time-consuming than labeling 2D images.

Self-supervised learning (SSL) aims to solve this by training models on "pretext tasks" that don't require human labels. However, existing 3D SSL methods often rely on contrastive learning, which requires complex data augmentations and massive GPU memory, or masking techniques that struggle with the inherent sparsity of Lidar data. The authors of ALSO sought a more direct, geometrically-grounded way to learn useful representations from raw, unannotated Lidar scans.

Methods

The core idea of ALSO is occupancy estimation. The intuition is simple: if a model can accurately reconstruct the underlying surface of a scene from sparse points, it must have captured meaningful information about the objects in that scene.

Overview of the ALSO approach, showing the transition from self-supervised pre-training to downstream tasks.

Key components include:

  • Visibility-based Supervision: Instead of human labels, the method uses the physics of the Lidar sensor. For every laser pulse that hits a surface, we know the space between the sensor and the hit point is empty. ALSO generates "query points" in this empty space and just behind the hit point (assumed "full") to create a free supervision signal.
  • Occupancy Decoder: A lightweight MLP takes a latent vector from the backbone and a query point's relative coordinates to predict if that point is occupied or empty.
  • "Semantization" through Neighborhood Reconstruction: Unlike standard reconstruction that aggregates many points to predict one, ALSO forces a single point's latent vector to be able to reconstruct its entire neighborhood. This encourages the backbone to learn more global, object-level features rather than just local geometry.
  • Single-Stream Pipeline: Unlike contrastive methods that process two versions of a scene, ALSO uses a single stream, allowing it to run on a modest 16GB GPU.
The query generation process leveraging sensor line-of-sight.

Datasets

The authors validated ALSO across a diverse set of benchmarks to prove its versatility:

  • nuScenes & SemanticKITTI: The "gold standards" for rotative Lidar segmentation and detection.
  • SemanticPOSS: Used to test cross-dataset generalization.
  • LivoxSimu: A synthetic dataset featuring non-rotative Lidar patterns (like the Livox Horizon), proving the method isn't tied to specific sensor hardware.
  • ONCE & KITTI3D: Large-scale datasets for evaluating 3D object detection performance.

Key Insights and Ideas

  • Geometry is a Proxy for Semantics: The paper demonstrates that a purely geometric pretext task (occupancy) naturally leads to semantic clustering in the latent space. t-SNE visualizations show that points belonging to the same class (e.g., cars vs. sidewalks) cluster together even before the model sees a single human label.
  • Simplicity Wins: By avoiding the "augmentation engineering" required for contrastive learning, ALSO is easier to implement and much less resource-intensive.
  • Sensor Agnosticism: Because it relies on basic visibility principles, it works across different Lidar technologies (rotative vs. solid-state) without modification.
  • Efficiency in Low-Label Regimes: The method is particularly powerful when very little annotated data is available (e.g., 0.1% or 1% of a dataset), providing a massive head start over training from scratch.

Key Results

ALSO consistently outperforms or matches state-of-the-art self-supervised methods:

  • Semantic Segmentation: On nuScenes and SemanticKITTI, ALSO outperformed PointContrast and SegContrast. For example, on SemanticKITTI with only 0.1% labels, ALSO achieved a 35.0 mIoU, a significant jump over the 30.0 mIoU of a non-pre-trained baseline.
  • Object Detection: The method is on par with specialized detection-focused SSL methods like ProposalContrast, despite ALSO being a more general-purpose geometric task.
  • Qualitative Superiority: Visual comparisons show that ALSO produces much cleaner, more coherent segmentations than competing SSL methods when fine-tuned on limited data.
Qualitative comparison showing ALSO's superior segmentation performance with limited training data.

Limitations and Future Work

  • Heuristic Assumptions: The assumption that space immediately behind a hit point is "full" is a heuristic. It can be wrong for very thin objects (like signs) or at grazing angles.
  • Sensor Metadata: The method requires knowing the sensor's location for each scan to calculate visibility, which might not be available in every legacy dataset.
  • Future Directions: The authors suggest that combining occupancy estimation with other SSL paradigms—like contrastive learning or masked autoencoders—could lead to even more robust representations. They also aim to explore how this geometric understanding can benefit temporal tasks like tracking.

Summary updated

Abstract

We propose a new self-supervised method for pre-training the backbone of deep perception models operating on point clouds. The core idea is to train the model on a pretext task which is the reconstruction of the surface on which the 3D points are sampled, and to use the underlying latent vectors as input to the perception head. The intuition is that if the network is able to reconstruct the scene surface, given only sparse input points, then it probably also captures some fragments of semantic information, that can be used to boost an actual perception task. This principle has a very simple formulation, which makes it both easy to implement and widely applicable to a large range of 3D sensors and deep networks performing semantic segmentation or object detection. In fact, it supports a single-stream pipeline, as opposed to most contrastive learning approaches, allowing training on limited resources. We conducted extensive experiments on various autonomous driving datasets, involving very different kinds of lidars, for both semantic segmentation and object detection. The results show the effectiveness of our method to learn useful representations without any annotation, compared to existing approaches. Code is available at https://github.com/valeoai/ALSO
BibTeX
@inproceedings{boulch2023also,
  author = {Alexandre Boulch and Corentin Sautier and Björn Michele and Gilles Puy and Renaud Marlet},
  title = {{ALSO: Automotive Lidar Self-supervision by Occupancy estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.01293},
}
Figure