Columbia University TRECVID-2006 Video Search and High-Level Feature Extraction

Shih-Fu Chang, Winston H. Hsu, Wei Jiang, Lyndon S. Kennedy, Dong Xu, Akira Yanagawa, Eric ZaveskyTRECVID 2006 · OpenReview

Abstract

This paper details Columbia University's approach to the TRECVID 2006 video search and high-level feature extraction tasks. By expanding concept lexicons from 39 to 374 categories and implementing context-aware fusion and query-specific reranking, the researchers achieved a 70% performance improvement over traditional text-only baselines. Key innovations include Lexicon-Spatial Pyramid Matching (LSPM) and query-class dependent fusion strategies.

Introduction

The "semantic gap"—the discrepancy between low-level visual features (pixels, color, texture) and high-level human concepts (e.g., "people marching")—remains a central challenge in video retrieval. In 2006, most systems were constrained by small concept lexicons, typically the LSCOM-lite set of 39 concepts. Columbia University’s TRECVID-2006 entry explored whether a massive expansion of this "visual vocabulary" to 374 concepts, combined with sophisticated contextual modeling and multimodal fusion, could significantly enhance search accuracy and robustness.

Methodology

The research focused on two primary tasks: High-Level Feature Extraction (HLFE) and Multimodal Search.

High-Level Feature Extraction

The baseline system utilized Support Vector Machines (SVMs) trained on standard visual features. To improve upon this, the team implemented several advanced techniques:

  • Context-Based Fusion: A Boosted Conditional Random Field (CRF) was used to model inter-conceptual relationships, refining detection scores based on correlated concepts (e.g., the presence of "outdoor" helping to identify a "car").
  • Lexicon-Spatial Pyramid Matching (LSPM): This novel method utilizes SIFT features and spatial pyramids to capture local object attributes, which is particularly effective for concepts defined by specific local details.
  • Event Detection: The team explored Earth Mover’s Distance (EMD) to analyze temporal dynamics across multiple frames within a shot rather than relying on a single keyframe.

Multimodal Search Pipeline

The search system employed a query-class dependent fusion strategy. Queries were categorized (e.g., "sports," "named entity," "concept"), and weights for text search, visual examples, and concept detectors were adjusted accordingly.

The multimodal search system architecture, illustrating the flow from query extraction to final fusion.

A key component of the pipeline is Information Bottleneck (IB) Reranking, a visual-based technique that promotes search results exhibiting recurrent visual patterns, effectively "pushing up" relevant content in the results list.

Key Results and Insights

The integrated system achieved a 70% improvement in Mean Average Precision (MAP) compared to the text-only baseline. Concept-based search emerged as the most significant contributor, providing a 30% relative boost over story-based text search.

Lexicon Scaling and LSPM

Expanding the lexicon from 39 to 374 concepts provided a 30% relative boost in search performance. While the increase in accuracy was not linear with the number of concepts, it greatly improved "query coverage"—the likelihood that a user's search term matches a pre-trained detector.

Performance comparison showing significant gains for specific concepts when using Lexicon-Spatial Pyramid Matching (LSPM).

As shown in the figure above, the LSPM method was particularly successful, yielding an 89% performance gain for the "waterscape waterfront" concept. Furthermore, for highly specific queries like "soccer goalposts," the concept-based approach outperformed baseline methods by 140%.

Discussion and Future Work

While the results were promising, the researchers noted limitations in event detection. The EMD-based approach for "people marching" did not yield improvements, likely due to overfitting on previous years' data and imprecise subshot boundaries. Additionally, the authors observed diminishing returns with very large lexicons, as many concepts suffer from poor detector quality due to sparse training data. Future research will focus on optimizing lexicon size and improving the robustness of individual detectors through more efficient fusion models.

Summary updated

BibTeX
@inproceedings{chang2006columbia,
  author = {Shih-Fu Chang and Winston H. Hsu and Wei Jiang and Lyndon S. Kennedy and Dong Xu and Akira Yanagawa and Eric Zavesky},
  title = {{Columbia University TRECVID-2006 Video Search and High-Level Feature Extraction}},
  booktitle = {TRECVID},
  year = {2006},
  url = {https://openreview.net/forum?id=h_dqV3qJ6Z7},
}
Figure