TL;DR
This paper introduces Temporal Overlapping Prediction (TOP), a novel self-supervised pre-training method for LiDAR Moving Object Segmentation (MOS). TOP leverages the inherent temporal inconsistencies in LiDAR scans to learn robust spatiotemporal representations, significantly reducing the reliance on expensive manual annotations for training MOS models.
What Problem Does This Paper Address?
Moving Object Segmentation (MOS) is a critical task for autonomous systems, enabling them to perceive and react to dynamic environments. However, existing supervised MOS methods require extensive manual annotation of LiDAR point clouds, which is both time-consuming and costly. While self-supervised learning offers a promising alternative, previous approaches often rely on complex tasks like forecasting future LiDAR scans. This forecasting can lead to learning noisy "flow" information, which may not transfer well to the downstream MOS task. This paper aims to bridge this gap by proposing a simpler, yet effective, self-supervised pre-training method that directly leverages temporal motion cues in LiDAR data for MOS.
Methods
The core of this paper is the Temporal Overlapping Prediction (TOP) method. TOP operates in two main stages:
- Temporal Overlapping Pre-processing: This stage identifies "temporal overlapping points." These are points in the current LiDAR scan that are also intersected by beams from adjacent (past or future) scans. The method accounts for the physical characteristic of LiDAR beam divergence, which can cause inconsistencies in occupancy measurements of the same point across different scans. It handles two scenarios based on the spatial angle between intersecting beams:
- Scenario-1 (Large Angle): The intersection is treated as a single point.
-
Scenario-2 (Small Angle): The intersection is treated as a region, and multiple points are sampled from it. The occupancy state of these temporal overlapping points is determined using a LiDAR occupancy measurement model that considers free, occupied, and unknown states based on confidence levels.
-
Self-supervised Pre-training Objectives: TOP employs two objectives to pre-train a LiDAR encoder (specifically, a sparse 4D UNet):
- Temporal Overlapping Prediction (L_overlap): The primary objective is to predict the occupancy state of the temporal overlapping points identified in the pre-processing step. This is achieved by feeding the positional encoding of the overlapping point and the feature of the corresponding current LiDAR point into a shallow MLP decoder. The loss function used is a weighted Cross-Entropy loss, with weights adjusted to handle class imbalance (more free points than occupied).
- Current Occupancy Reconstruction (L_recon): An auxiliary objective is introduced to reconstruct the occupancy of the current scan. This acts as a regularization term, helping to prevent overfitting and ensuring the learned representations are grounded in the current scene's geometry. This is also formulated as a Cross-Entropy loss.
The overall pre-training loss is a combination of these two objectives.
Datasets
- nuScenes: Used for pre-training and few-shot fine-tuning experiments. The paper notes that they refined MOS labels by incorporating object attributes and speed to better define motion states.
- SemanticKITTI: Used for cross-dataset transfer experiments.
Key Insights and Ideas
- Temporal Inconsistency as a Signal: The paper cleverly exploits the fact that the same physical point can appear to have different occupancy states across consecutive LiDAR scans due to sensor motion and object movement. This "temporal occupancy inconsistency" is the key signal for self-supervised learning.
- Avoiding "Scene Flow" Noise: By focusing on predicting occupancy states of overlapping points rather than full scene flow, TOP avoids learning potentially noisy and complex motion vectors, leading to more transferable representations.
- Beam Divergence Matters: Explicitly modeling beam divergence is crucial for accurately identifying temporal overlapping points and their occupancy states, preventing errors introduced by simplified beam models.
- Dual Objectives for Robustness: Combining temporal overlapping prediction with current occupancy reconstruction provides a balanced approach, ensuring both temporal understanding and geometric awareness are learned.
Key Results
- Superior Few-Shot Performance: TOP consistently outperforms training-from-scratch baselines and existing self-supervised pre-training methods (like LiDAR forecasting methods 4DOcc and UnO) on both object-level Recall (Recall_obj) and point-level IoU (IoU_w/o) metrics on the nuScenes dataset. For example, it achieved up to a 12.17% relative improvement in Recall_obj on the 10% data subset.
- Strong Cross-Dataset Transferability: TOP demonstrates excellent generalization capabilities, significantly outperforming baselines when fine-tuned on the SemanticKITTI dataset.
- Improved Semantic Segmentation: When evaluated on a few-shot semantic segmentation task on nuScenes, TOP also showed improved performance (0.42% mIoU increase at convergence) compared to training from scratch, suggesting its learned representations are broadly useful.
- Ablation Study: The paper shows that using both L_overlap and L_recon objectives leads to better performance than using L_overlap alone, indicating the benefit of the reconstruction term as a regularizer.
Limitations and Future Work
- Limitations: The paper acknowledges that the forecasting task is inherently more complex than MOS. While TOP aims to simplify this, the learned "flow" in forecasting methods can be noisy and may not transfer well. The paper also notes that the conventional IoU metric has biases (over-weighting large objects, including ego-vehicle points) which they address with their custom metrics.
- Future Work: The authors suggest investigating TOP for other temporal tasks, such as object tracking, as a promising avenue for future research. They also believe that exploring TOP for other temporal tasks beyond MOS is a promising direction.
Summary updated