Quality Aware Features for Performance Prediction and Time Reduction in Video Object Tracking

Roger Gomez Nieto, José Franciso Ruiz-Muñoz, Juan Berón, César A. Ardila Franco, Hernán Darío Benítez-Restrepo, Alan C. BovikIEEE Access 2022 · OpenReview

Abstract

Existing video object tracking (VOT) research often focuses on scene-dependent challenges like occlusion and clutter, frequently overlooking the impact of authentic in-capture and post-capture distortions. This paper introduces the Authentically Distorted Surveillance Videos (AD-SVD) dataset, comprising 4,476 videos featuring various distortions and surveillance activities. We propose a novel approach for performance prediction and quality-aware feature selection using 3D Convolutional Neural Networks (C3D) and Support Vector Regression (SVR). By predicting tracker performance across spatial resolutions, our framework enables a strategic reduction in computational resources—achieving up to a 34% reduction in execution time with only a marginal 3% decrease in tracking accuracy.

Introduction

Video Object Tracking (VOT) is a fundamental component of modern computer vision systems, particularly in the realm of automated surveillance. However, a significant gap exists between benchmark performance and real-world deployment. Most current VOT benchmarks prioritize scene-level challenges such as background clutter or target occlusion. In practice, surveillance footage is often plagued by "authentic distortions"—sensor-level issues like defocus, over-exposure, or low-light graininess that are not adequately represented in standard datasets.

State-of-the-art trackers are computationally expensive. Running these high-end algorithms at full resolution on distorted footage where they are likely to fail—or on clear footage where lower resolutions would suffice—is inefficient. This research addresses this by developing a system that predicts tracker performance before execution, allowing for intelligent resource management.

The AD-SVD Dataset

To bridge the gap in authentic distortion research, we introduced the Authentically Distorted Surveillance Videos (AD-SVD) dataset. This dataset is specifically designed to evaluate how trackers handle real-world sensor limitations rather than just environmental geometry.

Figure 1: An example of an outdoor surveillance scene from the AD-SVD dataset, representing the variety of environments captured.

Key characteristics of the dataset include:

  • Scale and Scope: 4,476 videos recorded across seven distinct indoor and outdoor locations.
  • Activity Diversity: Eight surveillance-specific activities, including fighting, robbery, and abandoned packages.
  • Distortion Focus: Systematic inclusion of in-capture distortions such as Defocus, Exposure (over and under), and Mixed distortions.
  • Benchmarking: We provided baseline results for 11 state-of-the-art trackers from the VOT 2017–2018 challenges, revealing that many top-tier trackers are surprisingly fragile when faced with simple lens blur.

Methodology: Performance Prediction and Optimization

Our framework utilizes a two-stage process to optimize tracking efficiency:

1. Quality-Aware Feature Extraction

Traditional video quality metrics often fail to correlate with the semantic needs of a tracking algorithm. Instead, we employ a 3D Convolutional Neural Network (C3D) to extract spatio-temporal features. These features are uniquely capable of capturing both the visual texture of distortions and the underlying motion dynamics of the scene.

2. SVR-Based Performance Mapping

These C3D features are fed into a Support Vector Regression (SVR) model. The model is trained to predict the expected Area Under Curve (AUC) of a tracker’s success plot. By analyzing the video chunks offline, the system can estimate the performance loss ($pl$) that would occur if the video were processed at a lower spatial resolution.

Figure 2: Indoor surveillance tracking example. The framework predicts whether tracking (indicated by the red bounding box) will remain successful if resolution is reduced.

Results and Resource Reduction

The primary innovation of this study is the ability to balance processing time and accuracy. We discovered a "Goldilocks" zone for many trackers: downscaling to 1/4 resolution typically yields significant speedups with negligible impact on the tracker's ability to maintain its target.

  • Efficiency Gains: The system achieved a 34% reduction in execution time with only a 3% decrease in performance across the dataset.
  • Extreme Optimization: For specific trackers like TFCR, we observed a 65% reduction in time with a marginal performance loss of only 0.03 AUC.
  • Prediction Accuracy: The C3D+SVR model demonstrated high reliability, with Pearson Linear Correlation Coefficient (PLCC) scores exceeding 0.90 in several test scenarios.

Conclusion

By distinguishing between scene-dependent challenges and sensor-dependent distortions, this research provides a more realistic benchmark for surveillance technology. The proposed performance prediction framework allows for a significant reduction in computational and storage requirements, making high-quality automated surveillance more sustainable for large-scale deployments. Future work will focus on adapting these tools for real-time, first-frame prediction and expanding the dataset to include environmental distortions like heavy rain and fog.

Summary updated

Abstract

The existing body of work on video object tracking (VOT) algorithms has studied various image conditions such as occlusion, clutter, and object shape, which influence video quality and affect tracking performance. Nonetheless, there is no clear distinction between the performance reduction caused by scene-dependent challenges such as occlusion and clutter, and the effect of authentic in-capture and post-capture distortions. Despite the plethora of VOT methods in the literature, there is a lack of detailed studies analyzing the performance of videos with authentic in-capture and post-capture distortions. We introduced a new dataset of authentically distorted videos (AD-SVD) to address this issue. This dataset contains 4476 videos with different authentic distortions and surveillance activities. Furthermore, it provides benchmarking results for evaluating ten state-of-the-art visual object trackers (from VOT 2017–2018 challenges) based on the proposed dataset. In addition, this study develops an approach for performance prediction and quality-aware feature selection for single-object tracking in authentically distorted surveillance videos. The method predicts the performance of a VOT algorithm with high accuracy. Then, the probability of obtaining the reference output is maximized without executing the tracking algorithms. We also propose a framework to reduce video tracker computation resources (time and video storage space). We achieve this by balancing processing time and tracking accuracy by predicting the performance in a range of spatial resolutions. This approach can reduce the execution time by up to 34% with a slight decrease in performance of 3%.
BibTeX
@inproceedings{nieto2022quality,
  author = {Roger Gomez Nieto and José Franciso Ruiz-Muñoz and Juan Berón and César A. Ardila Franco and Hernán Darío Benítez-Restrepo and Alan C. Bovik},
  title = {{Quality Aware Features for Performance Prediction and Time Reduction in Video Object Tracking}},
  booktitle = {IEEE Access},
  year = {2022},
  url = {https://openreview.net/forum?id=xD-7sCB0T98},
}
Figure