ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking

Nicola A. Piga, Yuriy Onyshchuk, Giulia Pasquale, Ugo Pattacini, Lorenzo Natale

Abstract

6D object pose tracking is a cornerstone of robotics and computer vision, yet most state-of-the-art solutions struggle with high-velocity motions. Methods leveraging deep neural networks often introduce non-negligible delays, causing tracking to degrade when objects move faster than the processing frame rate. We introduce ROFT, a Kalman filtering approach that leverages real-time optical flow to synchronize delayed outputs from low-frame-rate Convolutional Neural Networks (CNNs) for instance segmentation and pose estimation. By bridging this latency gap, ROFT achieves precise 6D pose and velocity tracking. Evaluated on the new Fast-YCB dataset and the HO-3D dataset, ROFT significantly outperforms existing methods in high-speed scenarios while maintaining real-time performance.

The Challenge of Latency in 6D Tracking

In reactive robotics—such as catching a thrown object or performing high-speed manipulation—knowing an object's 6D pose (position and orientation) and its 6D velocity is critical. While modern CNNs like Mask R-CNN or DOPE provide excellent single-frame estimates, they are computationally expensive. On standard hardware, these models often run at low frame rates (e.g., 5 fps), creating a "latency gap." If an object moves at high speed, it has already changed position by the time the CNN finishes processing a frame, leading to tracking lag or total failure. Furthermore, many existing trackers do not explicitly estimate 6D velocity, which is essential for smooth robotic control.

Methodology: Bridging the Gap with Optical Flow

ROFT addresses these challenges through a three-stage pipeline designed for synchronization and robustness:

  1. Optical Flow-Aided Segmentation: Since segmentation masks arrive with a delay, ROFT uses real-time optical flow to "warp" the last known mask forward in time. This ensures the mask remains synchronized with the current high-frame-rate RGB-D stream.
  2. 6D Velocity Tracking: The system employs a linear Kalman Filter to estimate linear and angular velocities. It maps 2D optical flow vectors to 3D motion using a specialized Jacobian matrix derived from the pinhole camera model and depth data, treating optical flow as a direct velocity sensor.
  3. 6D Pose Tracking & Outlier Rejection: An Unscented Kalman Filter (UKF) fuses estimated velocities with delayed pose measurements. To maintain accuracy, an outlier rejection step renders a synthetic depth map of the predicted pose and compares it to the actual depth sensor data; if the discrepancy is too high, the measurement is discarded.
The ROFT pipeline architecture, illustrating the synchronization of delayed CNN outputs with real-time optical flow and the outlier rejection mechanism.

Experimental Evaluation

To test the limits of tracking, the researchers introduced Fast-YCB, a photorealistic synthetic dataset featuring objects moving at speeds up to 63 cm/s and rotating at 266 deg/s. They also tested on HO-3D, a real-world dataset involving hand-held object manipulation.

Key Results:

  • Superior Accuracy: On Fast-YCB, ROFT achieved an average angular error of 12.6°, a massive improvement over PoseRBPF (42.9°) and se(3)-TrackNet (34.1°).
  • Robustness to Speed: Unlike competitors whose performance drops sharply as velocity increases, ROFT remains stable due to its integration of optical flow as a motion prior.
  • Real-Time Performance: ROFT operates at 96.2 fps on a mobile workstation, far exceeding the 30 fps requirement for real-time robotics, even when the underlying CNNs are significantly slower.
Qualitative comparison of ROFT against state-of-the-art trackers on the Fast-YCB (left) and real-world (right) sequences, showing ROFT's robustness to fast motion and occlusions.

Discussion and Future Work

The core innovation of ROFT is the realization that high-speed tracking doesn't necessarily require faster CNNs, but rather a robust way to propagate delayed results to the present. By using optical flow as a computationally cheap "bridge," ROFT maintains high-fidelity tracking even with slow sensors.

Current limitations include difficulties with "out-of-plane" rotations (where objects rotate toward or away from the camera) and the assumption of a stationary camera. Future iterations aim to incorporate ego-motion compensation to allow the tracker to function on moving robotic platforms or handheld devices.

Summary updated

Abstract

6D object pose tracking has been extensively studied in the robotics and computer vision communities. The most promising solutions, leveraging on deep neural networks and/or filtering and optimization, exhibit notable performance on standard benchmarks. However, to our best knowledge, these have not been tested thoroughly against fast object motions. Tracking performance in this scenario degrades significantly, especially for methods that do not achieve real-time performance and introduce non negligible delays. In this work, we introduce ROFT, a Kalman filtering approach for 6D object pose and velocity tracking from a stream of RGB-D images. By leveraging real-time optical flow, ROFT synchronizes delayed outputs of low frame rate Convolutional Neural Networks for instance segmentation and 6D object pose estimation with the RGB-D input stream to achieve fast and precise 6D object pose and velocity tracking. We test our method on a newly introduced photorealistic dataset, Fast-YCB, which comprises fast moving objects from the YCB model set, and on the dataset for object and hand pose estimation HO-3D. Results demonstrate that our approach outperforms state-of-the-art methods for 6D object pose tracking, while also providing 6D object velocity tracking. A video showing the experiments is provided as supplementary material.
BibTeX
@article{piga2022roft,
  author = {Nicola A. Piga and Yuriy Onyshchuk and Giulia Pasquale and Ugo Pattacini and Lorenzo Natale},
  title = {{ROFT: Real-Time Optical Flow-Aided 6D Object Pose and Velocity Tracking}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2022},
  url = {https://openreview.net/forum?id=kYakGugT6bh},
}
Figure