Low frame rate video target localization and tracking testbed

Yu Pang, Dan Shen, Genshe Chen, Pengpeng Liang, Khanh Pham, Erik Blasch, Zhonghai Wang, Haibin LingProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 2013 · DOI

TL;DR

This paper introduces the Saliency-based Temporal Association Dependency (STAD) framework to tackle the challenge of tracking objects in low frame rate video. By combining robust detection with temporal constraints and a confidence factor, STAD achieves stable and accurate tracking of multiple robots, outperforming several state-of-the-art methods.

What Problem Does This Paper Address?

Tracking objects in videos is a well-studied problem, but traditional methods often struggle when the video frame rate is low. In such scenarios, objects can move significantly between frames, leading to appearance changes and making it difficult for trackers to maintain their lock. This is particularly problematic for methods relying on smooth trajectories or frequent updates. The authors focus on this low frame rate (LFR) challenge, especially in the context of tracking multiple mobile robots in a controlled testbed environment. They highlight that existing trackers, even those designed for LFR like TLD, can fail due to appearance changes, illumination variations, and difficulties in distinguishing similar objects.

Methods

The STAD framework is a multi-stage approach designed for LFR tracking:

  • Background Subtraction for Initial Detection: A median filter is used to create a static background model. This model is then subtracted from incoming frames to identify potential foreground regions, providing a rough initial estimate of target locations.
  • Gradient-based Direction Detection: To overcome the limitations of simple template matching in LFR, the authors employ a novel marker design on the robots. These markers include auxiliary lines that facilitate a gradient voting algorithm. This allows for quick and accurate estimation of the robot's pose and direction, significantly narrowing down the search space for template matching.
  • Template Matching with Branch Pruning: Once the direction is estimated, a thresholding technique is applied to the candidate regions to create a binary representation. This is then matched against template images. To improve efficiency, a branch pruning technique is introduced, which uses integral images to quickly reject candidate regions that cannot possibly yield a better score, reducing processing time.
  • Temporal Dependency and Confidence Factor: To enhance stability and robustness, the framework incorporates temporal constraints. Information from previous frames (position and direction) is used to weigh the current candidate scores, assuming that targets don't change drastically between frames. A confidence factor is also introduced to mitigate the risk of incorrect previous information. This factor increases when a template is consistently matched and decreases if the score is poor or if scores for different templates are very similar, indicating uncertainty.
  • Association: Finally, the Hungarian algorithm is used to solve the assignment problem, ensuring a one-to-one correspondence between detected regions and templates, preventing multiple targets from being assigned to a single template.

Datasets

The paper introduces a custom robot testbed environment for evaluation. This testbed features multiple rigid mobile robots in a constrained area with a static background that can be affected by illumination changes. The images are captured using a wide-angle camera, introducing distortions, especially near the boundaries. The authors do not explicitly mention using publicly available datasets for training or evaluation, focusing instead on their specific testbed.

Key Insights and Ideas

  • Marker Design for Direction: The auxiliary lines on the robot markers are a key innovation, enabling efficient and accurate direction estimation which is crucial for LFR tracking. This significantly reduces the computational burden of traditional pose estimation.
  • Branch Pruning for Speed: The use of integral images for branch pruning is an effective way to speed up the template matching process by quickly discarding unlikely candidates.
  • Confidence Factor for Robustness: The confidence factor is a clever way to handle potential errors in temporal information. It allows the system to rely more on raw similarity scores when uncertain about previous frames, preventing cascading errors.
  • STAD's Advantage in LFR: The paper strongly argues that traditional trackers fail in LFR due to significant inter-frame motion. STAD's combination of robust detection, direction estimation, and temporal smoothing directly addresses these LFR-specific challenges.

Key Results

  • The STAD framework achieves near real-time performance, with a run time of approximately 0.1 seconds per frame after optimizations.
  • In their testbed, STAD demonstrates significantly better performance than Meanshift, Online-AdaBoosting (OAB), Multiple-Instance-Learning (MIL), and Tracking-Learning-Detection (TLD) in terms of failed frames and targets.
  • STAD is shown to be robust to targets moving out of and back into the scene, a common failure point for other methods.
  • The method can accurately track both the location and direction of targets.

Limitations and Future Work

  • Testbed Specificity: While the method shows strong results in their custom robot testbed, its generalizability to other complex, unconstrained environments with more diverse object types and motion patterns is not extensively demonstrated.
  • Boundary Issues: The paper acknowledges that tracking accuracy is severely affected when robots move near the camera boundaries due to wide-angle distortion.
  • Future Work: The authors suggest incorporating game-theoretical methods for multi-robot pursuit-evasion strategy comparisons as a future research direction.

Summary updated

Abstract

Traditional tracking frameworks are challenged by low video frame rate scenarios, because the appearances and locations of the target may change considerably in consecutive frames. Our paper presents a saliency-based temporal association dependency (STAD) framework to deal with such a low frame rate scenario and demonstrate good results in our robot testbed. We first use median filter to create a background of the scene, then apply background subtraction to every new frame to decide the rough position of the target. With the help of the markers on the robots, we use a gradient voting algorithm to detect the high responses of the directions of the robots. Finally, a template matching with branch pruning is used to obtain the finer estimation of the pose of the robots. To make the tracking-by-detection framework stable, we further introduce the temporal constraints using a previously detected result as well as an association technique. Our experiments show that our method can achieve a very stable tracking result and outperforms some state-of-the-art trackers such as Meanshift, Online-AdaBoosting, Mulitple-Instance-Learning, Tracking-Learning-Detection etc. Also. we demonstrate that our algorithm provides near real-time solutions given the low frame rate requirement.
BibTeX
@inproceedings{pang2013low,
  author = {Yu Pang and Dan Shen and Genshe Chen and Pengpeng Liang and Khanh Pham and Erik Blasch and Zhonghai Wang and Haibin Ling},
  title = {{Low frame rate video target localization and tracking testbed}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {2013},
  doi = {10.1117/12.2015954},
}
Figure