Adversarial Self-Supervised Scene Flow Estimation

Victor Zuanazzi, Joris van Vugt, Olaf Booij, Pascal Mettes

Introduction

Scene flow estimation is a fundamental task in 3D computer vision, particularly for autonomous systems. It involves predicting 3D motion vectors for every point in a sequence of point clouds, effectively serving as the 3D counterpart to optical flow. While supervised learning has shown promise, it is bottlenecked by the extreme difficulty of obtaining dense, point-level manual annotations for dynamic 3D scenes.

To address this, researchers have turned to self-supervised learning. However, most existing methods rely on a "correspondence assumption"—the belief that a point in one frame has an exact, identifiable match in the next. In real-world LiDAR applications, sensors re-sample the environment at every time step, meaning points are rarely sampled twice in the exact same location. This paper introduces a novel adversarial metric learning approach that eliminates the need for these unrealistic point-to-point correspondences.

Adversarial Metric Learning Framework

The core innovation of this work is an adversarial framework that operates in a learned latent space rather than on raw point coordinates. The system consists of two primary components: a Flow Extractor and a Cloud Embedder.

The adversarial training pipeline: A Flow Extractor predicts motion, while a Cloud Embedder maps results into a latent space for triplet-based metric learning.

Instead of minimizing point-level distances (like Chamfer distance), the model minimizes distances on a latent manifold. The training process utilizes several key strategies:

  • Multi-scale Triplet Loss: The target point cloud is partitioned into "anchor" and "positive" samples, while the flow-warped source cloud acts as the "negative" sample. This loss is applied across multiple network layers to capture both local geometry and global scene structure.
  • Adversarial Optimization: The Flow Extractor attempts to warp the source cloud to match the target's latent representation, while the Cloud Embedder acts as a discriminator, attempting to maximize the distance between warped and real clouds.
  • Cycle Consistency: To ensure temporal stability, the network is trained to predict the inverse flow when the sequence is reversed, using a combination of cosine similarity and L2 norms to prevent trivial zero-flow solutions.

The Scene Flow Sandbox

To rigorously evaluate scene flow models, the authors introduced the Scene Flow Sandbox. This benchmark consists of five datasets that scale in complexity from simple synthetic objects to unannotated real-world LiDAR data.

The Scene Flow Sandbox benchmark, ranging from single synthetic objects with full visibility to real-world LiDAR data from KITTI and Lyft.

The sandbox highlights a critical finding: models trained on data with perfect correspondences often fail when transitioned to re-sampled real-world data. By testing across these varying levels of partial visibility and data types (synthetic vs. real), the authors demonstrate the robustness of their latent-space approach.

Key Results and Insights

The proposed method achieves state-of-the-art performance in self-supervised settings. On the Multi ShapeNet dataset, it achieved an End Point Error (EPE) of 0.3497, a significant improvement over prior baselines (~0.82). On real-world KITTI data, the model reached an EPE of 0.5542 using a PointPWC-net backbone.

Qualitatively, the model excels at capturing the independent motion of multiple objects within a single scene. By avoiding explicit point matching, it maintains object geometry more effectively than correspondence-based methods, which often struggle with the noise inherent in LiDAR re-sampling.

Limitations and Future Work

Despite its success, the model faces challenges with occlusions. When an object leaves the field of view, the Flow Extractor occasionally attempts to merge its points with the nearest visible structure rather than predicting its exit from the scene. Additionally, in real-world data, the ego-motion of the sensor can make it difficult for the model to distinguish between moving objects and static backgrounds (like walls). Future research will focus on explicitly modeling partial observability to further bridge the gap between self-supervised and fully supervised scene flow estimation.

Summary updated

BibTeX
@inproceedings{zuanazzi2020adversarial,
  author = {Victor Zuanazzi and Joris van Vugt and Olaf Booij and Pascal Mettes},
  title = {{Adversarial Self-Supervised Scene Flow Estimation}},
  booktitle = {3DV},
  year = {2020},
  doi = {10.1109/3dv50981.2020.00115},
}
Figure