ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking

Nicola A. Piga, Yuriy Onyshchuk, Giulia Pasquale, Ugo Pattacini, Lorenzo Natale

TL;DR

ROFT (Real-time Optical Flow-aided Tracking) is a novel framework that achieves high-speed 6D object pose and velocity tracking by using real-time optical flow to "bridge the gap" between slow, delayed deep learning outputs and fast-moving camera streams. It significantly outperforms state-of-the-art trackers on high-velocity sequences, reducing angular errors by up to 71%.

What Problem Does This Paper Address?

In robotics, knowing exactly where an object is (6D pose) and how fast it’s moving (velocity) is crucial for tasks like catching or reactive manipulation. While deep learning models (CNNs) have become excellent at estimating pose from single frames, they are computationally heavy. On standard hardware, these models often run at low frame rates (e.g., 5 fps), introducing significant delays.

If an object is moving quickly, by the time a CNN finishes processing a frame, the object has already moved to a new position. This "latency gap" causes standard trackers to lag or lose the target entirely. Furthermore, most existing trackers don't explicitly estimate 6D velocity, which is vital for smooth, closed-loop robotic control.

Methods

The authors introduce a three-stage pipeline designed to handle these delays:

  1. Optical Flow-Aided Segmentation: Since instance segmentation masks (from Mask R-CNN) arrive delayed, the system uses real-time optical flow to "warp" the last known mask forward in time, keeping it synchronized with the current video frame.
  2. 6D Velocity Tracking: Using a linear Kalman Filter, the system estimates the object's linear and angular velocities. It does this by mapping 2D optical flow vectors to 3D motion using a specialized Jacobian matrix derived from the pinhole camera model and depth data.
  3. 6D Pose Tracking: An Unscented Kalman Filter (UKF) fuses the estimated velocities with delayed pose measurements (from a model like DOPE). To handle "bad" data, the system includes an outlier rejection step: it renders a synthetic depth map of the estimated pose and compares it to the actual depth sensor data; if they don't match, the measurement is discarded.

Datasets

The researchers evaluated ROFT on two main benchmarks:

  • Fast-YCB (New): A photorealistic synthetic dataset introduced in this paper. It features objects from the YCB set moving at high speeds (up to 63 cm/s linear and 266 deg/s rotational velocity).
  • HO-3D: A public dataset of real-world hand-held object manipulation, which includes challenging occlusions and complex rotations.

Key Insights and Ideas

  • Synchronization is Key: The "aha moment" is realizing that you don't need a faster CNN; you just need a way to propagate its old results to the present. Optical flow provides a computationally cheap way to do this.
  • Flow as a Velocity Sensor: Instead of just using optical flow for 2D tracking, the authors treat it as a direct measurement of 6D velocity. This allows the filter to have a much more accurate "motion prior" than simple constant-velocity models.
  • Verification via Rendering: By using the GPU to render what the object should look like at a predicted pose and comparing it to the real depth image, the system can autonomously detect when the CNN has provided a "hallucinated" or incorrect pose.

Key Results

  • Superior Accuracy: On the Fast-YCB dataset, ROFT achieved an average angular error of 12.6°, compared to 42.9° for PoseRBPF and 34.1° for se(3)-TrackNet.
  • Robustness to Speed: While other trackers' performance degraded sharply as object speed increased, ROFT remained stable due to its optical flow integration.
  • Real-Time Performance: ROFT runs at 96.2 fps on a mobile workstation, easily exceeding the 30 fps requirement for real-time robotics, even while the underlying CNNs only run at 5 fps.

Limitations and Future Work

  • Out-of-Plane Rotations: The system struggles when an object rotates "into" or "away from" the camera (out-of-plane), as optical flow can underestimate these motions due to self-occlusion.
  • Stationary Camera Assumption: The current version assumes the camera is static. Future work aims to incorporate "ego-motion" compensation so the tracker can function on a moving robot head or a handheld camera.
  • Covariance Adaptation: The authors suggest that automatically tuning the filter's "trust" levels (covariance matrices) based on tracking quality could further improve robustness.

Summary updated

Abstract

6D object pose tracking has been extensively studied in the robotics and computer vision communities. The most promising solutions, leveraging on deep neural networks and/or filtering and optimization, exhibit notable performance on standard benchmarks. However, to our best knowledge, these have not been tested thoroughly against fast object motions. Tracking performance in this scenario degrades significantly, especially for methods that do not achieve real-time performance and introduce non negligible delays. In this work, we introduce ROFT, a Kalman filtering approach for 6D object pose and velocity tracking from a stream of RGB-D images. By leveraging real-time optical flow, ROFT synchronizes delayed outputs of low frame rate Convolutional Neural Networks for instance segmentation and 6D object pose estimation with the RGB-D input stream to achieve fast and precise 6D object pose and velocity tracking. We test our method on a newly introduced photorealistic dataset, Fast-YCB, which comprises fast moving objects from the YCB model set, and on the dataset for object and hand pose estimation HO-3D. Results demonstrate that our approach outperforms state-of-the-art methods for 6D object pose tracking, while also providing 6D object velocity tracking. A video showing the experiments is provided as supplementary material.
BibTeX
@article{piga2022roft,
  author = {Nicola A. Piga and Yuriy Onyshchuk and Giulia Pasquale and Ugo Pattacini and Lorenzo Natale},
  title = {{ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  url = {https://openreview.net/forum?id=kYakGugT6bh},
}

Loading related content...

Figure