Glasgow University at TRECVid 2006

Jana Urban, Xavier Hilaire, Frank Hopfgartner, Robert Villa, Joemon M. Jose, Siripinyo Chantamunee, Yoshihiko GotohTRECVID 2006 · OpenReview

Introduction

As digital video libraries expanded exponentially in the mid-2000s, the primary challenge shifted from storage to retrieval. Researchers from the Universities of Glasgow and Sheffield participated in the TRECVid 2006 challenge to address the "semantic gap"—the disconnect between low-level visual features (like pixel color) and high-level human concepts (like "the ocean"). Their work focused on multi-modal fusion, combining Automatic Speech Recognition (ASR), Machine Translation (MT), and visual descriptors to improve search accuracy in massive broadcast news datasets.

Methodology

The research team employed a multi-pronged approach to video retrieval, focusing on both automatic and interactive search tasks.

Textual and Visual Retrieval

For textual retrieval, the team utilized the Terrier system with a BM25 model. A significant innovation was the implementation of a "shot window." Recognizing that spoken descriptions in news broadcasts often precede or follow the relevant visual footage, the system associated each shot with text from the six preceding and six following shots. This context-heavy approach aimed to mitigate alignment errors between audio and video.

For visual retrieval, five standard MPEG-7 descriptors were extracted: Color Layout, Contour Shape, Dominant Colors, Edge Histogram, and Homogeneous Texture. Shots were ranked based on visual similarity to query examples using Mahalanobis distance.

Graph-Based Fusion (ICG)

The study's most novel technical contribution was the Image-Context Graph (ICG). This model represented shots, terms, and visual features as nodes in a graph. By performing a "random walk" starting from query nodes, the system calculated the probability of reaching other shot nodes, effectively spreading relevance across different modalities.

Interactive Search and the MIR Tool

To facilitate human-in-the-loop retrieval, the team developed the MIR Tool, a custom graphical user interface. This tool allowed users to browse search results, play video clips, and provide explicit relevance feedback to refine the system's understanding of the query.

The MIR Tool interface used for interactive search tasks, featuring search, result, and playback panels.

Despite the tool's capabilities, researchers noted "interface friction." Users often found the relevance feedback loop slow and preferred marking final results over providing nuanced "maybe relevant" ratings.

Experimental Results

The evaluation was conducted using the TRECVid 2006 benchmark, which included broadcast news from Chinese and Arabic sources. The results highlighted the continued dominance of textual signals in video search.

Performance metrics for various automatic (UG_F) and interactive (UG_I) search runs, showing Mean Average Precision (MAP) and recall.

As shown in the results table, the interactive run (UG_I_1) was the top performer with a Mean Average Precision (MAP) of 0.047. The best automatic run was the text-only baseline (UG_F_1) with a MAP of 0.03. Visual-only runs (UG_F_2 and UG_F_5) performed poorly (MAP 0.004–0.005), confirming that global visual features are insufficient for general semantic queries but can be useful for specific topics with distinct visual signatures, such as sports or helicopters.

Discussion and Future Work

The study identified several bottlenecks in current video retrieval systems. Alignment errors between ASR/MT text and video content—often caused by anchor shots discussing future segments—remain a major hurdle. The authors suggested that future systems should move away from fixed "shot windows" toward smarter alignment based on speaker identification.

Furthermore, while the graph-based fusion model showed promise, it did not outperform the text baseline in official runs. Post-challenge analysis suggested that the model has significant untapped potential if parameters are tuned more effectively. Future iterations of the interactive interface should also allow users to select visual examples directly during the initial query to reduce friction.

Summary updated

BibTeX
@inproceedings{urban2006glasgow,
  author = {Jana Urban and Xavier Hilaire and Frank Hopfgartner and Robert Villa and Joemon M. Jose and Siripinyo Chantamunee and Yoshihiko Gotoh},
  title = {{Glasgow University at TRECVid 2006}},
  booktitle = {TRECVID},
  year = {2006},
  url = {https://openreview.net/forum?id=pIHit12Z4G},
}
Figure