VIREO/DVMM at TRECVID 2009: High-Level Feature Extraction, Automatic Video Search, and Content-Based Copy Detection

Chong-Wah Ngo, Yu-Gang Jiang, Xiao-Yong Wei, Wanlei Zhao, Yang Liu, Shiai Zhu, Jun Wang, Shih-Fu Chang

Introduction

The TRECVID 2009 campaign presented significant challenges in video analysis across three primary domains: High-Level Feature Extraction (HLFE), Automatic Video Search, and Content-Based Copy Detection (CBCD). This paper details the VIREO/DVMM team's participation, highlighting the introduction of Domain Adaptive Semantic Diffusion (DASD) to address the limitations of isolated concept detection and the use of adaptive fusion and geometric verification for search and copy detection tasks.

High-Level Feature Extraction via DASD

A core challenge in HLFE is the "domain shift" problem, where semantic relationships between concepts (e.g., "road" and "vehicle") vary across different video genres. To address this, the authors proposed Domain Adaptive Semantic Diffusion (DASD).

DASD models concepts as nodes in a graph, with edges representing their statistical affinity. Unlike static models, DASD employs a diffusion process to refine detection scores based on contextual evidence. For instance, a high-confidence detection of a "vehicle" can boost the score of a nearby, uncertain "road" detection. Crucially, the system adapts the graph's geometry online based on the test data, ensuring that the modeled relationships remain relevant to the specific video domain being processed.

Illustration of Domain Adaptive Semantic Diffusion (DASD) modeling inter-concept relationships.

Automatic Video Search and Flickr Context

For automatic search, the team moved beyond fixed-weight fusion of text and visual modalities. They implemented a "simulated query" approach to dynamically estimate the reliability of different search paths for specific query types.

A key innovation was leveraging external knowledge from Flickr. By analyzing metadata from 3.5 billion images, the team estimated concept affinities using Normalized Google Distance (NGD). This "Flickr Context Similarity" (FCS) provided a robust proxy for visual "common sense," outperforming purely linguistic ontologies like WordNet by capturing real-world visual co-occurrences.

Content-Based Copy Detection (CBCD)

The CBCD system utilized a Bag-of-Visual-Words (BoW) framework with SIFT features. To maintain high precision against transformations like cropping or re-encoding, the authors applied Enhanced Weak Geometric Consistency (EWGC) and a 2D Hough Transform. These techniques ensure that matched features between a query and a reference video maintain consistent spatial and temporal alignments, effectively filtering out false positives in large-scale datasets.

Experimental Results

Evaluation on the TRECVID 2009 benchmark demonstrated the efficacy of these approaches. In the HLFE task, DASD improved the Mean Average Precision (MAP) from a baseline of 0.156 to 0.163, showing gains in 17 out of 20 concepts.

Performance comparison of 222 system runs in TRECVID 2009, highlighting the impact of DASD and feature fusion.

In the search task, adaptive fusion yielded a 10% improvement over single-modality baselines. The CBCD system also achieved competitive results, particularly in re-encoding transformations (T3), reaching an Optimal Normalized Detection Cost Rate (Opt.NDCR) of 0.022.

Conclusion

The VIREO/DVMM participation in TRECVID 2009 underscores the importance of context and domain adaptation in video intelligence. While DASD and Flickr-driven search provided significant performance boosts, future work remains to improve localization accuracy in copy detection and to expand the vocabulary of reliable concept detectors to further bridge the semantic gap in video retrieval.

Summary updated

BibTeX
@inproceedings{ngo2009vireo,
  author = {Chong-Wah Ngo and Yu-Gang Jiang and Xiao-Yong Wei and Wanlei Zhao and Yang Liu and Shiai Zhu and Jun Wang and Shih-Fu Chang},
  title = {{VIREO/DVMM at TRECVID 2009: High-Level Feature Extraction, Automatic Video Search, and Content-Based Copy Detection}},
  booktitle = {TRECVID},
  year = {2009},
}
Figure