A Dataset for Semantic Segmentation of Point Cloud Sequences

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, Juergen GallarXiv 2019 · arXiv

Introduction

While deep learning has fundamentally transformed image-based semantic segmentation, 3D LiDAR data processing has historically faced significant hurdles. Early point cloud datasets were often limited in scale, restricted to indoor environments, or provided only isolated, static scans. For autonomous driving systems, however, a single snapshot is insufficient. Real-world navigation requires a continuous understanding of how a scene evolves over time and the ability to predict geometry in occluded areas.

To bridge this gap, the authors introduced SemanticKITTI, the first large-scale dataset to provide dense, point-wise semantic annotations for entire LiDAR sequences. By labeling all 22 sequences of the established KITTI Odometry Benchmark, this work provides a massive resource of over 4.5 billion points, enabling a shift from single-scan analysis toward comprehensive temporal scene understanding.

The SemanticKITTI Dataset

SemanticKITTI builds upon the foundation of the KITTI Vision Odometry Benchmark, significantly expanding its utility. The dataset comprises 23,201 scans for training and 20,351 for testing, covering a full 360° field-of-view for every scan.

The annotation effort resulted in 28 distinct semantic classes, ranging from common road participants like cars and bicycles to environmental features such as sidewalks, terrain, tree trunks, and poles. Compared to previous datasets like Oakland3D or Paris-Lille-3D, SemanticKITTI is substantially larger and uniquely offers sequential information, which is critical for training models to recognize temporal patterns.

Methodology and Labeling Pipeline

Creating a consistent dataset of this magnitude required a rigorous labeling pipeline. The authors utilized a SLAM (Simultaneous Localization and Mapping) system to register multiple scans within a sequence. By subdividing the environment into 100m x 100m tiles, annotators could label aggregated point clouds. This approach ensured temporal consistency—for instance, ensuring a "fence" identified in one scan remains labeled as a "fence" in subsequent frames as the vehicle moves past it.

A key innovation in the labeling process was the distinction between moving and non-moving objects. By observing behavior across a sequence, the authors could distinguish a car driving down the street from one parked at the curb—a distinction that is often impossible to make from a single LiDAR frame.

Baseline Architectures

The study evaluated several state-of-the-art approaches to establish performance benchmarks:

  • Point-based methods: Such as PointNet++, which process raw coordinates.
  • Graph-based methods: Such as SPGraph.
  • Projection-based methods: Such as SqueezeSeg, which project 3D data into 2D "range images."

The authors introduced two new baselines, DarkNet21Seg and DarkNet53Seg. These models apply efficient 2D convolutions to spherical projections of LiDAR data, leveraging the inherent "range image" structure produced by rotating LiDAR sensors.

Key Insights and Results

The evaluation of these baselines revealed several critical insights into the nature of 3D scene understanding:

The Sparsity Struggle

Model performance drops sharply as the distance from the sensor increases. Because LiDAR beams diverge with distance, distant objects are represented by very few points. This extreme sparsity makes it difficult for even the most advanced models to achieve high classification accuracy for far-off objects.

Projection vs. Point Clouds

One of the more surprising findings was that projection-based methods (like DarkNet53Seg) outperformed more complex point-based architectures. The authors attribute this to the higher parameter capacity of 2D CNNs and the efficiency of processing range images. DarkNet53Seg achieved a mean Intersection-over-Union (mIoU) of 49.9% on single scans. While this represents a strong baseline, it remains significantly lower than the ~80% mIoU typically seen in 2D image benchmarks like Cityscapes, highlighting the unique challenges of 3D data.

Semantic Scene Completion

The authors also introduced the task of Semantic Scene Completion, which requires models to predict both the geometry and semantics of occluded areas—essentially "imagining" what lies behind visible surfaces. This task proved exceptionally difficult, with the best-performing baseline reaching an mIoU of only 17.7%.

Conclusion and Future Work

SemanticKITTI has established itself as a cornerstone of LiDAR research, pushing the community toward a 4D understanding of urban environments. However, several challenges remain. Current models still struggle with long-range accuracy and have yet to fully exploit temporal data through recurrent neural networks or 3D convolutions. Furthermore, while the dataset provides class labels, the initial release did not distinguish between individual object instances. Despite these limitations, SemanticKITTI provides the necessary scale and complexity to drive the next generation of autonomous perception systems.

Summary updated

Abstract

Semantic scene understanding is important for various applications. In particular, self-driving cars need a fine-grained understanding of the surfaces and objects in their vicinity. Light detection and ranging (LiDAR) provides precise geometric information about the environment and is thus a part of the sensor suites of almost all self-driving cars. Despite the relevance of semantic scene understanding for this application, there is a lack of a large dataset for this task which is based on an automotive LiDAR. In this paper, we introduce a large dataset to propel research on laser-based semantic segmentation. We annotated all sequences of the KITTI Vision Odometry Benchmark and provide dense point-wise annotations for the complete $360^{o}$ field-of-view of the employed automotive LiDAR. We propose three benchmark tasks based on this dataset: (i) semantic segmentation of point clouds using a single scan, (ii) semantic segmentation using multiple past scans, and (iii) semantic scene completion, which requires to anticipate the semantic scene in the future. We provide baseline experiments and show that there is a need for more sophisticated models to efficiently tackle these tasks. Our dataset opens the door for the development of more advanced methods, but also provides plentiful data to investigate new research directions.
BibTeX
@misc{behley2019dataset,
  author = {Jens Behley and Martin Garbade and Andres Milioto and Jan Quenzel and Sven Behnke and Cyrill Stachniss and Juergen Gall},
  title = {{A Dataset for Semantic Segmentation of Point Cloud Sequences}},
  year = {2019},
  eprint = {1904.01416},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.01416},
}
Figure