BJTU TRECVID 2006 Video Retrieval System

Shikui Wei, Yao Zhao, Zhenfeng Zhu, Nan Liu, Yufeng Zhao, Liang Zhang, Fang WangTRECVID 2006

Abstract

This paper presents the video retrieval system developed by Beijing Jiaotong University for the TRECVID 2006 evaluation. The system addresses the "semantic gap" in video retrieval by combining a robust text-based automatic search engine with a novel interactive learning scheme based on the Co-Expectation-Maximization (Co-EM) algorithm. By partitioning features into independent visual and textual "views," the system leverages unlabeled data to significantly enhance retrieval precision with minimal user feedback. Evaluation on the TRECVID 2006 dataset demonstrates that the interactive approach substantially outperforms the initial automatic search, achieving high precision with as few as five positive user-labeled examples.

Introduction

The "semantic gap"—the disconnect between low-level digital features (like pixels and colors) and high-level human concepts (like "a person driving a car")—remains a primary hurdle in video retrieval. While interactive search, where a human provides feedback, can help bridge this gap, most existing systems suffer from two flaws: they require a large amount of manually labeled data, and they often treat different modalities (text vs. visual) in isolation. The BJTU system aims to create a system that provides a strong initial "automatic" result and then refines it through an interactive process that maximizes the utility of unlabeled data.

Methodology

The system operates in two distinct phases: an automatic "starter" engine and a Co-EM interactive scheme.

Automatic Search Engine

To provide users with an initial set of results, the system employs a text-based search engine. It processes Automatic Speech Recognition (ASR) transcripts and machine-translated text using a standard NLP pipeline involving tokenization, stop-word removal (460 terms), and Porter stemming. The retrieval model is implemented via the Lemur toolkit, utilizing a KL-divergence model to score the similarity between user queries and shot transcripts.

Co-EM Interactive Scheme

The core innovation of the system is the Co-EM interactive scheme. Once the automatic engine provides initial results, the user labels a few positive examples. The system then applies the Co-EM algorithm, a hybrid of multi-view learning and probabilistic EM.

The feature space is divided into two approximately independent views:

  • Feature A (Visual): A 36-dimensional HSV color histogram representing global color distribution.
  • Feature B (Textual): A 78-dimensional vector created by measuring the similarity of a shot's transcript to 78 "proto-concepts" derived from the LSCOM ontology.

Two SVM classifiers (one for each view) are trained iteratively. Crucially, they "teach" each other: Classifier A labels unlabeled data to help train Classifier B, and vice versa. To optimize performance, the system maps new queries to "proto-topics" to select optimal SVM parameters discovered through prior grid-searches.

The Co-EM interactive learning architecture and parameter selection process.

Results and Discussion

The automatic search proved surprisingly effective, achieving median or better performance for most topics in the TRECVID 2006 benchmark. However, the Co-EM interactive approach provided a substantial performance boost. By labeling only five positive examples, the interactive search achieved a precision significantly higher than the automatic search across the top 200 returned results.

Comparison of average precision at different ranks (P@X) between automatic and interactive search.

The system demonstrates that unlabeled data is a valuable resource; instead of ignoring the thousands of shots the user didn't label, Co-EM treats them as a pool of information that helps the classifiers converge on the user's intent. By explicitly splitting visual and textual features, the system can find relevant videos even when one modality is sparse or generic.

Conclusion

The BJTU TRECVID 2006 system successfully leverages multi-view learning to bridge the semantic gap in video retrieval. The integration of Co-EM allows for high-precision results with minimal human intervention. Future work will explore more sophisticated, query-dependent fusion methods and deeper visual descriptors to further improve retrieval accuracy.

Summary updated

BibTeX
@inproceedings{wei2006bjtu,
  author = {Shikui Wei and Yao Zhao and Zhenfeng Zhu and Nan Liu and Yufeng Zhao and Liang Zhang and Fang Wang},
  title = {{BJTU TRECVID 2006 Video Retrieval System}},
  booktitle = {TRECVID},
  year = {2006},
}
Figure