Abstract
Training robots to perform complex manipulation tasks from human demonstrations—known as Imitation Learning—requires policies that can handle high-dimensional action spaces and multimodal data distributions. While Diffusion Policies have recently established a new standard for these tasks, they often involve complex iterative noise processes and high computational costs. This paper introduces PointFlowMatch, a novel framework that replaces standard diffusion models with Conditional Flow Matching (CFM). By combining this efficient generative framework with 3D point cloud observations, PointFlowMatch achieves a 67.8% success rate on the RLBench benchmark, effectively doubling the performance of previous state-of-the-art methods.
Introduction
In robotic imitation learning, a policy must learn to map sensory observations to precise sequences of actions. This is challenging because human demonstrations are inherently multimodal; for instance, there are many valid trajectories to pick up a cup or open a drawer. While Diffusion Policies have shown promise in capturing these distributions, they rely on a complex "forward noise" process and can be computationally expensive during real-time inference.
PointFlowMatch addresses these limitations by leveraging Conditional Flow Matching (CFM), a simpler and more flexible generalization of diffusion. Instead of reversing a noise process, CFM learns a "velocity field" that transforms a simple distribution (Gaussian noise) into the target action distribution along straight-line paths. When paired with 3D spatial data, this approach provides a more robust and efficient solution for robotic manipulation.
Methodology
The PointFlowMatch architecture is built upon three core pillars designed to maximize spatial awareness and generative efficiency.
1. Conditional Flow Matching (CFM)
Unlike traditional diffusion models that require complex schedules to reverse noise, CFM focuses on learning the probability paths between distributions. It learns a vector field that defines the "velocity" required to move a sample from a noise distribution toward the expert action distribution. Because these paths are designed to be straight lines in the probability space, the model is mathematically simpler and often more efficient to solve using standard ODE solvers.
2. Point Cloud Backbone
Rather than relying on 2D RGB images, which can be sensitive to lighting and camera perspective, PointFlowMatch utilizes 3D point clouds. These are processed through a modified PointNet architecture to extract spatial features. This 3D representation naturally encodes the geometry of the scene, which is critical for tasks requiring high precision and spatial reasoning.
3. Action Generation and Rotation Formulation
The policy outputs a 10-dimensional action vector consisting of 3D position, 6D rotation, and a 1-bit gripper state. The core of the generation process is a 1D U-Net that predicts the velocity needed to move a noisy action trajectory toward a successful expert trajectory.
A key technical contribution of the study is the investigation of Rotation Formulation. The authors compared regressing rotations in standard Euclidean space against regressing them directly on the $SO(3)$ manifold using geodesic interpolation. Surprisingly, Euclidean regression performed better. The authors identified a "No Free Lunch" scenario where manifold-aware paths create mathematical discontinuities at the "opposite pole" of the rotation, making it significantly harder for the neural network to learn a smooth, continuous mapping.
Experimental Results
Benchmark Performance
The authors evaluated PointFlowMatch on RLBench, a large-scale benchmark for robot learning, across eight diverse tasks (e.g., unplug_charger, open_fridge, put_books_on_shelf).
- PointFlowMatch: 67.8% average success rate.
- OL-ChainedDiffuser (Previous SOTA): 34.6%.
- Diffusion Policy (Image-based): 18.7%.
The transition from RGB images to point clouds alone accounted for a performance jump of approximately 27 percentage points, highlighting the importance of 3D geometric data in manipulation.
Inference Efficiency
One of the primary advantages of CFM is its performance at low iteration counts. While CFM and Diffusion (DDIM) perform similarly at 50 steps, CFM maintains significantly higher accuracy when restricted to 1 to 5 inference steps. This makes PointFlowMatch a superior candidate for real-time robotics where high frequency (FPS) is required.
Real-World Validation
The model was deployed on a physical Franka Emika Panda robot to verify its utility outside of simulation. In real-world trials, the system achieved a 72% success rate for opening a box and 48% for placing a sponge on a plate, demonstrating that the simulation-trained policies generalize effectively to physical hardware.
Discussion and Limitations
While PointFlowMatch sets a new benchmark, it is not without limitations. Like other generative models that require iterative ODE solving, it possesses higher inference latency compared to simple "one-shot" behavior cloning models. Additionally, the reliance on point clouds necessitates high-quality depth cameras and precise extrinsic calibration to merge multiple viewpoints effectively.
Future work may involve integrating CFM into hierarchical models that plan high-level "keypoints" before executing low-level actions, potentially extending the framework's success to even longer-horizon, multi-stage tasks.
Summary updated