MFTIQ: Multi-Flow Tracker with Independent Matching Quality Estimation

Jonas Serych, Michal Neoral, Jiri MatasWACV 2025

TL;DR

This paper introduces MFTIQ, a novel dense long-term point tracking method that improves upon the Multi-Flow Tracker (MFT) by decoupling optical flow estimation from correspondence quality assessment. This "plug-and-play" approach allows MFTIQ to integrate with any off-the-shelf optical flow method, achieving state-of-the-art accuracy while being significantly faster than previous methods.

What Problem Does This Paper Address?

Tracking arbitrary points within video sequences, known as point-level visual tracking, is crucial for applications like 3D reconstruction and augmented reality. While bounding box or segmentation-based tracking is common, tracking individual points offers finer-grained information. Traditional methods often rely on chaining optical flow (OF) estimates between consecutive frames. However, this sequential approach suffers from error accumulation, leading to trajectory drift and failure in the presence of occlusions. Existing methods like Multi-Flow Tracker (MFT) attempt to address this by considering temporally distant frames and estimating flow chain reliability. However, MFT tightly couples quality estimation with OF computation and can be overly pessimistic in selecting shorter, less accurate flow chains. This paper aims to improve the accuracy, flexibility, and efficiency of dense long-term point tracking by decoupling quality estimation from OF and enabling seamless integration with various OF methods.

Methods

The core innovation of MFTIQ lies in its Independent Quality (IQ) module. Unlike MFT, which estimates occlusion and correspondence quality jointly with optical flow, MFTIQ separates these estimations. This decoupling allows for direct estimation of quality for the entire optical flow chain between the template and current frames, avoiding error-prone uncertainty accumulation.

MFTIQ builds upon the flow-chaining concept of MFT. Given a sequence of frames, it constructs candidate optical flow chains by combining previously computed flows (from template to an intermediate frame) with newly computed flows (from the intermediate to the current frame). These chains are formed for logarithmically spaced time deltas (e.g., 1, 2, 4, 8 frames apart, and direct match).

The Independent Quality (IQ) estimation network takes the chained optical flow, the template frame, and the current frame as input. It uses image features extracted by a pre-trained network (DINOV2) and combines them with a local correlation cost volume to estimate two crucial maps: a cost map (E), analogous to MFT's uncertainty, indicating positional accuracy, and an occlusion map (O), indicating whether a point is occluded or out-of-view. These maps are then used to select the most reliable flow chain per pixel.

The selection criterion for the best flow chain is based on a weighted combination of the cost map and the occlusion map, ensuring that unoccluded chains are always preferred. This "plug-and-play" design means MFTIQ can be trained once and then used with any off-the-shelf optical flow method without requiring fine-tuning or architectural modifications.

Datasets

  • TAP-VID DAVIS [10]: Used for evaluating point tracking performance. It contains 30 videos with semi-automatically annotated points and visibility states.
  • POT-210 [34]: A planar object tracking dataset used to evaluate MFTIQ's performance on homography estimation. It includes 210 videos with challenging scenarios like motion blur, occlusion, and perspective distortion.
  • Kubric rendering tool [19]: Used for generating synthetic training data. This dataset includes 200 sequences with variable numbers of static and dynamic objects, rendered at high resolution and with long sequence lengths.

Key Insights and Ideas

  • Decoupling Quality from Flow is Key: Separating the estimation of correspondence quality and occlusion from the optical flow computation itself significantly improves accuracy and flexibility. This allows for more robust quality assessment of the entire flow chain.
  • "Plug-and-Play" for Optical Flow: MFTIQ's ability to work with any off-the-shelf optical flow method is a major advantage. This allows users to leverage the latest advancements in OF research without needing to retrain the entire tracking system.
  • Improved Flow Chain Selection: The IQ module leads to a more conservative and accurate selection of flow chains compared to MFT, which often favored shorter chains with less reliable direct optical flows. MFTIQ selects chains that lead to better accuracy, even if they involve more intermediate frames.
  • Faster Dense Tracking: Despite achieving comparable or better accuracy than state-of-the-art trackers, MFTIQ offers significantly faster processing speeds for dense tracking tasks.

Key Results

  • On the TAP-VID DAVIS dataset, MFTIQ with the ROMA optical flow method achieved the best position accuracy (<davg) and second-best overall score (AJ), outperforming the original MFT and many other state-of-the-art trackers.
  • MFTIQ demonstrated significantly faster processing speeds compared to other trackers with similar accuracy, especially in dense tracking scenarios.
  • On the POT-210 dataset, MFTIQ with ROMA and RANSAC-based homography estimation achieved state-of-the-art performance, particularly excelling in scenarios with motion blur and scale changes.
  • Qualitative results showed MFTIQ producing fewer false re-detections than MFT, especially in challenging scenarios with occlusions and out-of-view objects.

Limitations and Future Work

  • Occlusion Accuracy: While MFTIQ improves upon MFT, its occlusion accuracy (OA) is noted as being comparatively lower, which affects the overall AJ score. Achieving state-of-the-art occlusion accuracy remains an open challenge.
  • Causality: Unlike some sparse point trackers that can "see into the future" (non-causal), MFTIQ, like MFT, is causal, only using previous frames. This is a limitation for tasks where future information is beneficial.
  • Future Work: The authors anticipate that future advancements in optical flow technology will further enhance MFTIQ's performance due to its "plug-and-play" nature. The architecture's compatibility with evolving OF techniques promises continued improvements in tracking precision and computational efficiency.

Summary updated

BibTeX
@inproceedings{serych2025mftiq,
  author = {Jonas Serych and Michal Neoral and Jiri Matas},
  title = {{MFTIQ: Multi-Flow Tracker with Independent Matching Quality Estimation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure