Abstract
6D object pose tracking is a cornerstone of robotics and computer vision, yet most state-of-the-art solutions struggle with high-velocity motions. Methods leveraging deep neural networks often introduce non-negligible delays, causing tracking to degrade when objects move faster than the processing frame rate. We introduce ROFT, a Kalman filtering approach that leverages real-time optical flow to synchronize delayed outputs from low-frame-rate Convolutional Neural Networks (CNNs) for instance segmentation and pose estimation. By bridging this latency gap, ROFT achieves precise 6D pose and velocity tracking. Evaluated on the new Fast-YCB dataset and the HO-3D dataset, ROFT significantly outperforms existing methods in high-speed scenarios while maintaining real-time performance.
The Challenge of Latency in 6D Tracking
In reactive robotics—such as catching a thrown object or performing high-speed manipulation—knowing an object's 6D pose (position and orientation) and its 6D velocity is critical. While modern CNNs like Mask R-CNN or DOPE provide excellent single-frame estimates, they are computationally expensive. On standard hardware, these models often run at low frame rates (e.g., 5 fps), creating a "latency gap." If an object moves at high speed, it has already changed position by the time the CNN finishes processing a frame, leading to tracking lag or total failure. Furthermore, many existing trackers do not explicitly estimate 6D velocity, which is essential for smooth robotic control.
Methodology: Bridging the Gap with Optical Flow
ROFT addresses these challenges through a three-stage pipeline designed for synchronization and robustness:
- Optical Flow-Aided Segmentation: Since segmentation masks arrive with a delay, ROFT uses real-time optical flow to "warp" the last known mask forward in time. This ensures the mask remains synchronized with the current high-frame-rate RGB-D stream.
- 6D Velocity Tracking: The system employs a linear Kalman Filter to estimate linear and angular velocities. It maps 2D optical flow vectors to 3D motion using a specialized Jacobian matrix derived from the pinhole camera model and depth data, treating optical flow as a direct velocity sensor.
- 6D Pose Tracking & Outlier Rejection: An Unscented Kalman Filter (UKF) fuses estimated velocities with delayed pose measurements. To maintain accuracy, an outlier rejection step renders a synthetic depth map of the predicted pose and compares it to the actual depth sensor data; if the discrepancy is too high, the measurement is discarded.
Experimental Evaluation
To test the limits of tracking, the researchers introduced Fast-YCB, a photorealistic synthetic dataset featuring objects moving at speeds up to 63 cm/s and rotating at 266 deg/s. They also tested on HO-3D, a real-world dataset involving hand-held object manipulation.
Key Results:
- Superior Accuracy: On Fast-YCB, ROFT achieved an average angular error of 12.6°, a massive improvement over PoseRBPF (42.9°) and se(3)-TrackNet (34.1°).
- Robustness to Speed: Unlike competitors whose performance drops sharply as velocity increases, ROFT remains stable due to its integration of optical flow as a motion prior.
- Real-Time Performance: ROFT operates at 96.2 fps on a mobile workstation, far exceeding the 30 fps requirement for real-time robotics, even when the underlying CNNs are significantly slower.
Discussion and Future Work
The core innovation of ROFT is the realization that high-speed tracking doesn't necessarily require faster CNNs, but rather a robust way to propagate delayed results to the present. By using optical flow as a computationally cheap "bridge," ROFT maintains high-fidelity tracking even with slow sensors.
Current limitations include difficulties with "out-of-plane" rotations (where objects rotate toward or away from the camera) and the assumption of a stationary camera. Future iterations aim to incorporate ego-motion compensation to allow the tracker to function on moving robotic platforms or handheld devices.
Summary updated