RSF: Optimizing Rigid Scene Flow From 3D Point Clouds Without Labels

David Deng, Avideh ZakhorWACV 2023

TL;DR

This paper introduces RSF, a novel method for estimating 3D scene flow from point clouds without requiring any ground truth labels. By modeling scene flow as a global ego-motion combined with rigid motions of detected objects (represented by bounding boxes), RSF achieves state-of-the-art performance on challenging benchmarks like KITTI Scene Flow and nuScenes, outperforming even supervised methods.

What Problem Does This Paper Address?

Estimating 3D scene flow – essentially, a 3D motion field for all points in a scene – is crucial for robotics and autonomous driving. While deep learning has advanced this field, most methods rely on expensive, often synthetic, annotations that don't generalize well to real-world LiDAR data. A key limitation of existing approaches is their failure to exploit the inherent rigidity of dynamic scenes, where objects typically move as coherent, rigid bodies. Instead, many methods predict unconstrained, point-wise motion vectors, leading to physically inconsistent and inaccurate results. This paper tackles the challenge of estimating accurate and physically plausible scene flow from unlabeled 3D point clouds by leveraging the multi-body rigidity prior.

Methods

The core innovation of RSF lies in its object-level rigid scene flow parameterization. Instead of predicting individual point movements, RSF models scene flow as a combination of:

  • Global Ego-Motion ($T_{ego}$): The overall motion of the sensor platform.
  • Object-Level Rigid Motions: A set of bounding boxes, each representing a potentially moving object. Each bounding box has its own rigid transformation ($T_i$) and a confidence score ($c_i$) indicating whether it contains a moving object.

To optimize these parameters without labels, RSF introduces a novel loss function based on the nearest neighbor distance (NND) between transformed points from the first point cloud ($P_1$) and the second point cloud ($P_2$). A key technical challenge is making the bounding box selection differentiable. The authors achieve this by proposing a differentiable bounding box approximation. This is done by replacing hard step functions with sigmoid approximations for the box's membership function along each axis, allowing gradients to flow through the bounding box parameters.

The overall loss for a single bounding box combines a foreground loss (NND after applying the object's rigid motion and ego-motion) and a background loss (NND after applying only ego-motion), weighted by the box's confidence score. This encourages the confidence score to converge towards 1 for dynamic objects and 0 for static ones. Additional auxiliary terms are used for regularization:

  • Box Dimension Regularization ($L_{shape}$): Constrains bounding box dimensions to realistic car-like sizes.
  • Heading Term ($L_{heading}$): Encourages the bounding box's heading to align with its motion direction.
  • Angle Term ($L_{angle}$): Penalizes large rotations for bounding boxes.
  • Mass Term ($L_{mass}$): Encourages bounding boxes to enclose more points, improving robustness.

The entire scene flow is then optimized by minimizing this combined loss function using gradient-based optimization, without any neural networks for prediction.

Datasets

RSF is evaluated on the following datasets:

  • KITTI Scene Flow: A real-world autonomous driving dataset with 142 point cloud pairs and ground truth scene flow and instance segmentation annotations. The authors note that pedestrians and cyclists are removed, leaving only cars.
  • nuScenes: A more challenging dataset with sparser LiDAR sweeps and complex scenarios. A subset released by [25] is used.
  • SemanticKITTI: Used for evaluating ego-motion prediction and motion segmentation accuracy.

The paper also references Flying Things3D (FT3D) and RawKITTI as datasets used by prior supervised and self-supervised methods for training and comparison.

Key Insights and Ideas

  • Leveraging Rigidity is Key: The paper strongly argues that exploiting the multi-body rigidity inherent in dynamic scenes is crucial for accurate, label-free scene flow estimation. This is a significant departure from point-wise prediction methods.
  • Differentiable Bounding Boxes Unlock Optimization: The development of a differentiable bounding box formulation is a critical enabler for optimizing object-level scene flow parameters using gradient descent. This allows the system to learn object boundaries and motions simultaneously.
  • Confidence Score as a Detector: The confidence score ($c_i$) learned for each bounding box effectively acts as a moving object detector, allowing the system to distinguish between dynamic objects and static background.
  • Joint Optimization for Ego-Motion and Object Motion: By jointly optimizing ego-motion and object-specific rigid motions, the method benefits from shared information and improved robustness, especially for static background elements.

Key Results

RSF achieves state-of-the-art performance on multiple benchmarks without any labeled supervision:

  • KITTI Scene Flow: Achieves 2x lower EPE3D than previous state-of-the-art methods when using the entire point cloud on StereoKITTI. Even with downsampling, it outperforms other methods.
  • nuScenes: Achieves an average error of 0.107 m, significantly outperforming [25].
  • Motion Segmentation: Significantly outperforms the baseline [1] in both mIoU and accuracy.
  • Ego-Motion Estimation: Outperforms ICP on SemanticKITTI, achieving an average rotation error of 0.235° and translation error of 0.107 m.
  • Qualitative Results: Visualizations show RSF accurately predicting flow on moving cars, even in challenging LiDAR settings, and effectively segmenting objects with bounding boxes.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Not Real-Time: As an optimization framework, RSF is not suitable for real-time inference.
  • Driving-Specific Parameterization: The current formulation is tailored for autonomous driving, where objects can be reasonably parameterized by boxes.
  • Challenges with Sparse/Featureless Regions: The method still struggles with sparse, occluded, or featureless portions of point clouds, particularly in LiDAR data.

Future work includes combining RSF with learning-based methods for real-time inference and generalizing the approach to arbitrary 3D scenes beyond the driving domain.

Summary updated

BibTeX
@inproceedings{deng2023rsf,
  author = {David Deng and Avideh Zakhor},
  title = {{RSF: Optimizing Rigid Scene Flow From 3D Point Clouds Without Labels}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure