THU-IMG at TRECVID 2009

Yingyu Liang, Binbin Cao, Jianmin Li, Chenguang Zhu, Yongchao Zhang, Chenhao Tan, Ge Chen, Chen Sun, Jinhui Yuan, Mingxing Xu, Bo ZhangTRECVID 2009 · OpenReview

Introduction

As digital video content continues to proliferate, the challenges of managing and identifying large-scale datasets become increasingly critical. The THU-IMG team from Tsinghua University addressed these challenges during the TRECVID 2009 competition by developing robust systems for two primary tasks: Content-Based Copy Detection (CBCD) and High-Level Feature Extraction (HLFE). Their approach emphasizes a hierarchical processing pipeline that balances computational efficiency with the high precision required to handle complex visual distortions, such as picture-in-picture transformations.

Content-Based Copy Detection (CBCD)

The core of the THU-IMG copy detection strategy is a hierarchical "video-only" system designed to process queries in stages. This architecture allows the system to quickly identify easy matches while reserving more intensive resources for difficult cases.

Hierarchical Architecture

The system consists of two primary subsystems:

  1. Block-Based Subsystem: This initial stage utilizes Block Gradient Histograms (BGH). To specifically address the "picture-in-picture" (PIP) problem, the system extracts BGH features from 16 distinct sub-windows of varying sizes and positions. A decision logic based on a ScoreTimes threshold determines if the result is conclusive. If the top-ranked match is significantly stronger than the second, the process terminates early for efficiency.
  2. Keypoint-Based Subsystem: If the block-based stage is inconclusive, the system invokes a more detailed analysis using SURF (Speeded Up Robust Features). This stage employs a "Vocabulary-Based Hashing" (VBH) index, which integrates Bag-of-Features with Locality Sensitive Hashing (LSH) to maintain high retrieval speeds despite the complexity of keypoint matching.
The hierarchical architecture of the video-only copy detection system, showing the transition from block-based to keypoint-based analysis.

Temporal Alignment and Fusion

For temporal matching, the team improved upon standard consistency methods by using the sum of logarithms of weights. This mathematical adjustment favors paths with several highly significant frame matches over those with many low-quality, coincidental matches. Regarding multi-modal fusion, the team found that different task profiles required different logic: an "AND" strategy (requiring both audio and video matches) was best for minimizing false alarms, while an "OR" strategy was preferred for balanced performance.

High-Level Feature Extraction (HLFE)

The HLFE system was built as a massive ensemble designed to "understand" semantic concepts within video frames. The pipeline involved:

  • Feature Diversity: Extraction of 36 different feature types, including color, Gabor textures, gradients, and Canny edges.
  • Spatial Context: Features were extracted across multiple region schemes (e.g., 4x3 grids and various splits) to capture localized semantic information.
  • Learning and Fusion: The team trained five SVMs for each feature pair using a boosting method. Ultimately, a simple sum fusion of these region-based features proved more effective than more complex weighted fusion attempts.

Experimental Results

The systems were evaluated using TRECVID 2009 benchmarks, with parameters tuned on the 2007 and 2008 datasets.

Copy Detection Performance

The hierarchical system performed consistently above the TRECVID average. The specialized sub-windowing approach for PIP transformations (Transformation 2) was particularly successful, yielding results significantly higher than the mean participant performance.

Optimal mean F1 scores for true positives across different transformation types, demonstrating the system's robustness.

Feature Extraction Performance

In the HLFE task, the Canny Edge Histogram emerged as the strongest single low-level feature (MAP of 0.0322). However, the power of the ensemble was evident in the final results; the best-performing run achieved a Mean Average Precision (MAP) of 0.134 through simple sum fusion.

Conclusion and Future Work

The THU-IMG team demonstrated that a hierarchical approach—combining fast block-based features with robust keypoint analysis—is highly effective for large-scale video copy detection. While the system excelled at handling spatial distortions like PIP, the authors noted that the impact of audio features remained limited. Future research will focus on automating parameter optimization and developing more sophisticated audio-visual integration to further enhance robustness and scalability.

Summary updated

BibTeX
@inproceedings{liang2009thu,
  author = {Yingyu Liang and Binbin Cao and Jianmin Li and Chenguang Zhu and Yongchao Zhang and Chenhao Tan and Ge Chen and Chen Sun and Jinhui Yuan and Mingxing Xu and Bo Zhang},
  title = {{THU-IMG at TRECVID 2009}},
  booktitle = {TRECVID},
  year = {2009},
  url = {https://openreview.net/forum?id=ZZJUeLWSF3},
}
Figure