TL;DR
This paper introduces MFT, a novel dense, pixel-level long-term tracking method that leverages multiple optical flow estimates with varying time differences. By intelligently selecting the most reliable flow chain based on learned occlusion and uncertainty scores, MFT achieves state-of-the-art performance on challenging benchmarks while being significantly faster than existing methods.
What Problem Does This Paper Address?
Tracking points consistently over long video sequences is a notoriously difficult problem in computer vision. Traditional methods often rely on concatenating optical flow estimates between consecutive frames. While simple, this approach suffers from significant drawbacks: error accumulation leads to "drift," and it fails to recover trajectories after occlusions. Another approach, matching every frame to a reference frame, is robust to occlusions but struggles with significant changes in pose and illumination. The core challenge is to maintain accurate, dense correspondences over extended periods, even in the presence of occlusions and appearance changes, without accumulating excessive drift.
Methods
The proposed method, MFT (Multi-Flow Tracker), tackles this by going beyond simple frame-to-frame optical flow. MFT computes optical flows not only between consecutive frames but also for pairs of frames with logarithmically spaced time differences (e.g., ∆=1, ∆=2, ∆=4, etc.). This creates multiple "candidate chains" of flows, each representing a potential trajectory.
The key innovation lies in how MFT selects the best candidate chain for each pixel. This selection is guided by two small, pre-trained Convolutional Neural Networks (CNNs) that estimate:
- Occlusion: A binary score indicating whether a pixel is occluded in the current frame.
- Uncertainty: A measure of the confidence in the estimated optical flow.
These CNNs operate on the cost-volume and feature representations used by the underlying optical flow network (specifically, RAFT in their implementation). The occlusion and uncertainty maps are then "chained" along with the optical flow estimates for each candidate ∆. Finally, for each pixel, MFT selects the candidate chain with the lowest uncertainty among those not predicted to be occluded. If all candidates are occluded, the first one is chosen. This strategy allows MFT to dynamically switch between direct (∆=∞) and chained (∆=1, 2, etc.) flows, effectively mitigating drift and handling occlusions.
Datasets
MFT was evaluated on the following datasets:
- TAP-Vid DAVIS and TAP-Vid Kinetics: These datasets, introduced for sparse point tracking, were adapted for dense tracking by using their provided annotations.
- BADJA: This benchmark was used to evaluate MFT on animal joint tracking.
The authors also mention using Sintel, Flying Things, and Kubric for training their occlusion and uncertainty estimation CNNs.
Key Insights and Ideas
- Chaining flows with varying time differences is crucial for long-term tracking: By considering flows over different temporal scales, MFT can adapt to scene dynamics and avoid the limitations of single-frame or fixed-interval approaches.
- Learned occlusion and uncertainty are powerful selection criteria: Instead of relying on heuristic methods, using CNNs to predict occlusion and flow uncertainty provides a robust mechanism for selecting the most reliable trajectory. This is a significant departure from many existing optical flow methods that ignore or poorly handle occlusions.
- The "best" flow chain is pixel-dependent: MFT's ability to select a different ∆ for each pixel allows it to adapt to local image conditions, choosing direct flows for stable regions and chained flows for areas prone to drift or short occlusions.
- Input resolution significantly impacts optical flow performance: The paper demonstrates that upscaling low-resolution videos, even without adding new details, substantially improves the accuracy of state-of-the-art optical flow methods like RAFT.
Key Results
- Competitive Performance: MFT achieved competitive performance on the TAP-Vid benchmark, outperforming baselines by a significant margin.
- Speed Advantage: MFT is orders of magnitude faster than other dense pixel-level tracking methods, running at 2.3 FPS on a single GPU.
- Interactive Speed: With pre-computed optical flows, MFT can achieve over 100 FPS, making it suitable for interactive applications like video editing.
- Robustness: The method is insensitive to medium-length occlusions and robustified against drift due to its reference frame tracking strategy.
- BADJA Performance: MFT showed competitive performance on the BADJA animal joint tracking benchmark.
Limitations and Future Work
- Spurious Re-detections: The authors acknowledge that MFT can sometimes match out-of-view parts of the template to visually similar regions in the current frame. This can lead to incorrect re-detections and affect the rest of the tracklet.
- Failure on Thin Structures: As seen in Figure 5, MFT (and the underlying RAFT flow) struggles with very thin structures like animal legs, which are difficult to resolve with current optical flow techniques.
- Future Work: The paper suggests that the learned occlusion and uncertainty estimation, combined with the multi-flow chaining strategy, could be extended to other dense prediction tasks. The authors also highlight the significant impact of input resolution on optical flow accuracy, suggesting further research into resolution-aware optical flow methods.
Summary updated