Let It Flow: Simultaneous Optimization of 3D Flow and Object Clustering

Patrik Vacek, David Hurych, Karel Zimmermann, Tomás Svoboda

TL;DR

This paper introduces a novel self-supervised framework for 3D scene flow estimation that jointly optimizes point motion and object clustering. By combining overlapping "soft" clusters with progressively growing "hard" clusters, the method achieves state-of-the-art accuracy on LiDAR datasets—particularly for challenging small objects like pedestrians—while running up to 10x faster than previous optimization-based competitors.

What Problem Does This Paper Address?

Estimating 3D scene flow—the three-dimensional motion vector of every point in a LiDAR scan—is vital for autonomous driving tasks like collision avoidance and object tracking. However, obtaining ground-truth labels for this motion is incredibly difficult and expensive.

Current self-supervised methods try to solve this by assuming the world is made of rigid objects. They typically use a two-step process: first, cluster points into "objects" (using algorithms like DBSCAN), and second, optimize the flow so that each cluster moves rigidly. This approach often falls into two traps:

  1. Under-segmentation: A pedestrian walking near a building might be lumped into the "static" cluster, causing the algorithm to ignore the pedestrian's motion.
  2. Over-segmentation: A single car might be split into multiple clusters, leading to inconsistent motion estimates across the vehicle.

Furthermore, many state-of-the-art methods rely on "Neural Priors" (using a neural network as a regularizer), which the authors argue can over-smooth motion and struggle to distinguish multiple independent objects moving close to one another.

Methods

The authors propose a "Neural-Prior-free" optimization approach that uses three distinct loss functions to balance local rigidity with global scene structure:

  • Hard Rigid Clusters: These are non-overlapping, small clusters that are assumed to belong to a single rigid body. As the optimization progresses, these clusters "grow" and merge if their estimated flows are consistent.
  • Soft Rigid Clusters: These are fixed-size, overlapping neighborhoods. Unlike hard clusters, they allow for outlier rejection. The method uses spectral clustering (specifically, the principal eigenvector of a reward matrix) to determine which points in a neighborhood actually belong to the same rigid motion.
  • The "Spring" Analogy: The authors visualize their loss functions as a mechanical system. A distance loss acts like a spring pulling points toward their likely positions in the next frame, while rigidity losses act like internal springs maintaining the shape of clusters.

By optimizing these together, the flow field and the object segmentation refine each other simultaneously.

Datasets

The method was evaluated on four major autonomous driving benchmarks:

  • Argoverse 1 & 2: Large-scale LiDAR datasets with complex urban maneuvers.
  • Waymo Open Dataset: A standard benchmark for high-quality LiDAR perception.
  • StereoKITTI: A classic dataset where 3D points are "lifted" from stereo images.

Key Insights and Ideas

  • Moving Beyond the Neural Prior: While Neural Priors are popular, the authors found they act as a bottleneck in highly dynamic scenes. Direct optimization of flow, guided by the right clustering constraints, proved more expressive and accurate.
  • The Power of Overlap: By using overlapping soft clusters, the algorithm can "bridge" the gap between different rigid parts. If a point is an outlier in one cluster but an inlier in another, the system can still resolve its motion correctly.
  • Handling the "Long Tail": Most scene flow metrics are dominated by static backgrounds and large trucks. This paper specifically highlights improvements in "long-tail" classes—pedestrians and cyclists—where previous methods often failed due to under-segmentation.

Key Results

  • Accuracy: The method reached new state-of-the-art results for self-supervised flow. On the StereoKITTI benchmark, it achieved a 38% improvement in End-Point Error (EPE) compared to the previous top-performer (SCOOP).
  • Class-Specific Gains: On Argoverse 2, the method significantly outperformed competitors on dynamic objects. For pedestrians, it achieved an EPE of 0.031m, compared to 0.062m for the popular NSFP method.
  • Efficiency: Because the soft cluster computations are performed on fixed-size neighborhoods, they can be highly parallelized on a GPU. This allowed the method to run 10x faster than MBNSFP while reaching the same level of error.

Limitations and Future Work

  • Optimization Speed: While faster than many competitors, the method is still an optimization-based approach performed at "inference" time, meaning it is not yet fast enough for real-time, high-frequency vehicle control.
  • Local Minima: Like most self-supervised methods relying on distance transforms or Chamfer loss, the optimization can occasionally get stuck in local minima if the initial ego-motion compensation is poor.
  • Future Directions: The authors suggest that integrating this clustering logic into a learning-based (feed-forward) network could combine the speed of deep learning with the structural robustness of their current optimization framework.

Summary updated

Abstract

We study the problem of self-supervised 3D scene flow estimation from real large-scale raw point cloud sequences. The problem is crucial to various automotive tasks like trajectory prediction, object detection, and scene reconstruction. In the absence of ground truth scene flow labels, contemporary approaches concentrate on deducing and optimizing flow across sequential pairs of point clouds by incorporating structure-based regularization on flow and object rigidity. The rigid objects are estimated by a variety of 3D spatial clustering methods. While state-of-the-art methods successfully capture overall scene motion using the Neural Prior structure, they encounter challenges in discerning multi-object motions. We identified the structural constraints and the use of large and strict rigid clusters as the main pitfall of the current approaches, and we propose a novel clustering approach that allows for a combination of overlapping soft clusters and non-overlapping rigid clusters. Flow is then jointly estimated with progressively growing non-overlapping rigid clusters together with fixed-size overlapping soft clusters. We evaluate our method on multiple datasets with LiDAR point clouds, demonstrating superior performance over the self-supervised baselines and reaching new state-of-the-art results. Our method excels in resolving flow in complicated dynamic scenes with multiple independently moving objects close to each other, including pedestrians, cyclists, and other vulnerable road users.
BibTeX
@article{vacek2025let,
  author = {Patrik Vacek and David Hurych and Karel Zimmermann and Tomás Svoboda},
  title = {{Let It Flow: Simultaneous Optimization of 3D Flow and Object Clustering}},
  journal = {IEEE Trans. Intell. Veh.},
  year = {2025},
  doi = {10.1109/tiv.2024.3443316},
}
Figure