Non-rigid target tracking based on 'flow-cut' in pair-wise frames with online hough forests

Tao Zhu, Yanning Zhang, Peng Zhang, Wei Huang, Hichem SahliACM Multimedia 2013 · DOI

TL;DR

This paper introduces "Flow-Cut," a novel method for tracking non-rigid objects by improving foreground segmentation using motion information between consecutive frames. By integrating this refined segmentation into an online learning framework with Hough Forests, the authors achieve significantly more accurate and robust tracking of deformable objects compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

Tracking non-rigid objects, like a person performing a complex action or a deformable piece of clothing, is notoriously difficult due to their constant changes in shape, pose, and appearance. Traditional tracking methods often rely on fixed shapes (like bounding boxes) or simple segmentation techniques that struggle with these intrinsic variations. While some methods try to improve segmentation by using multiple patches or pre-defined object parts, they often fail to consider the crucial temporal information between frames. This lack of temporal context makes accurate segmentation challenging, especially when the background resembles the target, leading to poor tracking performance. This paper aims to bridge this gap by leveraging motion information to achieve more precise segmentation, which in turn provides better training data for robust online tracking.

Methods

The core innovation is the "Flow-Cut" model for foreground segmentation. It builds upon a Gibbs energy function that combines two key components:

  • Appearance Model: This part uses a standard approach to model the target's appearance based on pixel intensity distributions (histograms) learned from labeled foreground and background regions.
  • Smoothness Energy: This is where the novelty lies. It incorporates:
  • Spatial Smoothness: Encourages neighboring pixels within the same frame to have similar segmentation labels (e.g., both foreground or both background).
  • Motion Smoothness (Flow-Cut): This is the key contribution. It uses the optical flow between consecutive frames to enforce smoothness on the segmentation boundary. By estimating the displacement field (flow) between frames, the model ensures that the segmented contour remains consistent with the object's motion, leading to more accurate boundary convergence.

This improved segmentation is then integrated into a semi-supervised online tracking framework. The segmented foreground regions are used to generate positive training samples for an online Hough Forest classifier, while background regions provide negative samples. Hough Forests are chosen for their robustness to noisy labels and their ability to handle variations in appearance. The framework updates the classifier online as the object moves, allowing it to adapt to appearance changes.

Datasets

The paper evaluates its method on "benchmark video sequences" that contain both rigid and non-rigid objects with challenging intrinsic deformations and occlusions. Specific dataset names are not provided, but the authors mention using sequences that are commonly used for evaluating tracking algorithms.

Key Insights and Ideas

  • Motion is Key for Non-Rigid Segmentation: The paper strongly demonstrates that incorporating motion information (optical flow) between frames is crucial for accurately segmenting deformable objects, especially when backgrounds are similar to the target.
  • Precise Segmentation = Better Training Data: The improved segmentation accuracy directly translates to higher quality training samples for the online classifier, leading to more robust tracking.
  • Hough Forests for Robust Online Learning: The use of Hough Forests in conjunction with the refined segmentation provides a powerful mechanism for online learning, capable of handling noisy data and adapting to appearance changes.
  • 'Flow-Cut' as a General Improvement: The proposed 'Flow-Cut' segmentation method offers a significant improvement over single-frame segmentation techniques and even over methods that use rough guidance like 'GrabCut' with Hough Forests.

Key Results

  • Improved Accuracy and Robustness: The proposed "Flow-Cut" tracker shows significant improvements in tracking accuracy and robustness compared to state-of-the-art methods like HT (Hough-based Tracking), BHMC (Basin Hopping Monte-Carlo Sampling), and TLD (Tracking-Learning-Detection).
  • Quantitative Performance: Table 1 shows the percentage of correctly tracked frames until failure. The "Flow-Cut" tracker achieves an average of 76.1%, outperforming HT (62.2%), BHMC (48.6%), TLD (53.4%), and CT (54.0%).
  • Qualitative Performance: Figure 1 visually demonstrates the superior performance of "Flow-Cut" in challenging scenarios, showing it maintaining track on objects where other methods fail.
  • Speed: The average running speed is reported as 1.5-2.5 fps without code optimization, which is reasonable for real-time applications.

Limitations and Future Work

  • Long-Term Challenges: The authors acknowledge that while their method handles presented problems well, there is still room for improvement in tackling more challenging scenarios such as long-term heavy occlusions, abrupt motion, and significant appearance changes.
  • Future Directions: These challenging problems are identified as areas for future research and improvement.

Summary updated

BibTeX
@inproceedings{zhu2013non,
  author = {Tao Zhu and Yanning Zhang and Peng Zhang and Wei Huang and Hichem Sahli},
  title = {{Non-rigid target tracking based on 'flow-cut' in pair-wise frames with online hough forests}},
  booktitle = {ACM Multimedia},
  year = {2013},
  doi = {10.1145/2502081.2502130},
}
Figure