The NTU Toolkit and Framework for High-Level Feature Detection at TRECVID 2007

Ming-Fang Weng, Chun-Kang Chen, Yi-Hsuan Yang, Rong-En Fan, Yu-Ting Hsieh, Yung-Yu Chuang, Winston H. Hsu, Chih-Jen LinTRECVID 2007 · OpenReview

Abstract

Researchers from National Taiwan University (NTU) developed a specialized toolkit that reduces SVM training time for video concept detection from two weeks to just three days. By combining optimized training with the "recycling" of past detectors and novel post-processing techniques—specifically concept reranking and temporal filtering—they achieved a 46% performance boost in High-Level Feature (HLF) detection.

Introduction

Semantic concept detection, the identification of high-level features such as "airplane," "sports," or "desert" in video shots, is fundamental to video indexing. However, the field faces significant challenges: high computational costs for training Support Vector Machines (SVMs), the scarcity of annotated data, and "context blindness" where models ignore temporal continuity and inter-concept relationships. This paper presents a three-pillar framework designed to overcome these hurdles through optimized training, model aggregation, and context-aware post-processing.

Methodology

The NTU framework consists of three primary components: an optimized toolkit, a model aggregation strategy, and a post-processing pipeline.

The NTU Toolkit

The team modified the standard LIBSVM library to optimize it specifically for high-level feature detection. Key improvements include:

  • Dense Inner Products: Optimized code for dense feature vectors halved computational time.
  • Parallelism: Simultaneous processing of multiple concept-feature pairs pushed CPU utilization above 90%.
  • Narrowed Search: Reducing the parameter grid search from 110 to 42 points accelerated the process without compromising quality.

Post-processing and System Architecture

The post-processing framework exploits both inter-concept and inter-shot relationships to refine initial detections.

The NTU high-level feature detection framework architecture.
  • Concept Reranking: This utilizes "inter-concept" relationships. For instance, high confidence in "Water" can be used to boost the score of a "Boat" detector in the same shot.
  • Temporal Filtering: This addresses "inter-shot" relationships. Since concepts like "Weather" often span multiple shots, predictions are smoothed using a weighted combination of neighboring shots determined by statistical tests.

Aggregation of Past Detectors

Rather than training solely on new data, the authors employed "late aggregation" to combine TRECVID 2007 detectors with models trained on 2005–2006 data. They utilized Concept-dependent Weighted Aggregation, where weights are learned based on the performance of older detectors on a validation set.

Experimental Results

The framework was evaluated on the TRECVID 2007 benchmark, utilizing data from 2005 and 2006 as "past data." The NTU toolkit reduced training time from 14 days to just 3 days, enabling more iterative experimentation.

Performance comparison of various HLF runs.

As shown in the results table, the best-submitted run (A_ntTransformer_5) achieved an infMAP of 0.0874, representing a 46% improvement over the baseline (A_nt20Giants_6). Specific gains were observed for contextually related concepts: reranking improved "Charts" by 45% and "Boat/Ship" by 29%, while temporal filtering boosted "Weather" detection by 37%.

Conclusion

The NTU framework demonstrates that significant gains in video concept detection can be achieved through efficiency and context. By "recycling" old models and applying temporal and contextual post-processing, the researchers improved accuracy without requiring massive new datasets. Future work will explore the parallel combination of reranking and filtering to further enhance performance.

Summary updated

BibTeX
@inproceedings{weng2007ntu,
  author = {Ming-Fang Weng and Chun-Kang Chen and Yi-Hsuan Yang and Rong-En Fan and Yu-Ting Hsieh and Yung-Yu Chuang and Winston H. Hsu and Chih-Jen Lin},
  title = {{The NTU Toolkit and Framework for High-Level Feature Detection at TRECVID 2007}},
  booktitle = {TRECVID},
  year = {2007},
  url = {https://openreview.net/forum?id=In6BofTWywL},
}
Figure