Prioritizing test cases for deep learning-based video classifiers

Yinghua Li, Xueqi Dang, Lei Ma, Jacques Klein, Tegawendé F. BissyandéEmpir. Softw. Eng 2024 · DOI

TL;DR

This paper introduces VRank, the first test prioritization technique specifically designed for deep learning-based video classifiers. By leveraging temporal, embedding, prediction, and uncertainty features, VRank effectively prioritizes video test cases that are most likely to reveal model faults, significantly improving efficiency and bug detection compared to existing methods.

What Problem Does This Paper Address?

Deep learning models are increasingly used for video classification across various applications. However, testing these models is challenging due to the sheer volume and temporal nature of video data, making manual labeling of test cases expensive and time-consuming. Existing test prioritization techniques, often developed for images or text, fail to account for the unique temporal dynamics of videos. This gap leads to inefficient testing, delaying the identification of bugs and hindering the development of reliable video classification systems.

Methods

VRank tackles this problem by proposing a novel test prioritization approach that specifically addresses the characteristics of video data. The core idea is to predict the probability of a video test case being misclassified by a deep neural network (DNN) classifier. This prediction is achieved by:

  1. Video-oriented Feature Generation: VRank extracts four types of features from each video test input:
  2. Temporal Features (TF): Captures the temporal coherence by analyzing variations and statistical properties between consecutive frames.
  3. Video Embedding Features (EF): Represents the intrinsic information of the video by averaging frame embeddings extracted using a pre-trained model (ResNet).
  4. Prediction Features (PF): Reflects the DNN classifier's confidence in its prediction for the video, derived from the output probabilities.
  5. Uncertainty Features (UF): Quantifies the model's uncertainty using various established metrics (e.g., DeepGini, Vanilla SM).
  6. Learning-to-Rank: A ranking model (LightGBM in the primary approach) is trained using these feature vectors. The model learns to predict a misclassification score for each test input.
  7. Test Prioritization: Test inputs are ranked in descending order based on their predicted misclassification scores, with higher scores indicating a higher priority for labeling and testing.

The paper also explores variants of VRank using different ranking models like XGBoost, Random Forest, and Logistic Regression.

Datasets

The study utilizes three widely-adopted video datasets:

  • HWID12: Real-world surveillance videos for highway accident detection.
  • HMDB51: Video clips sourced from movies, covering various daily actions.
  • UCF101: Action recognition dataset collected from YouTube.

To evaluate VRank's robustness, noisy versions of these datasets were generated using seven noise generation techniques: Channel Shift (CSR), Feature-wise Normalization (FSN), Height Shift (HS), Width Shift (WS), Shear (SR), Horizontal Flip (HF), and ZCA Whitening (ZCA).

Key Insights and Ideas

  • Temporal Information is Crucial: The paper strongly emphasizes that video data's temporal nature is a key differentiator from images and text, and any effective prioritization method must account for this.
  • Feature Fusion for Better Prioritization: Combining diverse feature types (temporal, embedding, prediction, uncertainty) provides a more comprehensive understanding of a video's potential to reveal model flaws than any single feature type alone.
  • Proximity to Decision Boundary: The underlying principle that inputs closer to the model's decision boundary are more likely to be misclassified is a guiding concept, and VRank's features are designed to capture this proximity.
  • Noise Robustness Matters: VRank demonstrates significant effectiveness even when applied to noisy video data, highlighting its practical applicability in real-world scenarios where data quality can be compromised.

Key Results

  • Effectiveness on Natural Datasets: VRank achieved an average improvement of 5.76% to 46.51% in effectiveness (measured by APFD) compared to existing test prioritization methods (DeepGini, VanillaSM, PCS, Entropy) and random selection.
  • Effectiveness on Noisy Datasets: VRank maintained its superiority on noisy datasets, showing an average improvement of 4.26% to 53.56%.
  • Efficiency: The feature generation step takes approximately 2.3 minutes, ranking model training about 35 seconds, and prediction less than 1 second, resulting in an overall execution time of around 3 minutes per dataset, which is considered acceptable given the cost of manual labeling.
  • Feature Contribution: An ablation study showed that all four feature types (TF, EF, PF, UF) contribute to VRank's effectiveness, with no single feature type being overwhelmingly dominant.
  • Number of Frames: Increasing the number of extracted frames (from 4 to 32) slightly improved VRank's effectiveness, but the gains diminished, suggesting a trade-off between effectiveness and efficiency. The authors settled on 16 frames.

Limitations and Future Work

  • Dependency on Visual Features: The current VRank implementation focuses solely on visual features, neglecting audio information. Future work will incorporate audio analysis to provide a more holistic understanding of video content.
  • Contextual Understanding: VRank's current ability to capture broader contextual aspects and narrative structures within videos is limited. Future research aims to explore methods for capturing narrative structures to enhance contextual understanding.
  • Whole Video Classification: The study focused on multi-class datasets where each video (and thus each frame) belongs to a single category. Future work might explore scenarios with more complex labeling.
  • Generalizability: While diverse datasets and models were used, the generalizability to entirely different domains or model architectures might be a future research direction.

Summary updated

BibTeX
@article{li2024prioritizing,
  author = {Yinghua Li and Xueqi Dang and Lei Ma and Jacques Klein and Tegawendé F. Bissyandé},
  title = {{Prioritizing test cases for deep learning-based video classifiers}},
  journal = {Empir. Softw. Eng.},
  year = {2024},
  doi = {10.1007/s10664-024-10520-1},
}
Figure