The MediaMill TRECVID 2007 Semantic Video Search Engine

Cees G. M. Snoek, Ivo Everts, Jan C. van Gemert, Jan-Mark Geusebroek, Bouke Huurnink, Dennis Koelma, M. van Liempt, Ork de Rooij, Koen E. A. van de Sande, Arnold W. M. Smeulders, Jasper R. R. Uijlings, Marcel WorringTRECVID 2007 · OpenReview

Abstract

The MediaMill team’s TRECVID 2007 entry tackles the "semantic gap" in video retrieval by scaling up to a massive 572-concept lexicon and introducing the ForkBrowser, a novel interactive tool. By combining multi-level visual features (grid, keypoints, and blobs) with a robust late-fusion strategy, they achieved top-tier rankings (3rd and 4th) in interactive search tasks, proving that more concepts and smarter browsing can significantly reduce user effort.

Introduction: Bridging the Semantic Gap

Searching for video is inherently difficult because most commercial engines rely on text-based metadata like filenames, tags, or transcripts. This approach fails when the visual content is not explicitly described or when the audio quality is poor. To bridge this "semantic gap," researchers utilize concept detectors—automated tools that recognize visual entities like "desert," "car," or "meeting."

However, as the number of detectable concepts grows from dozens to hundreds, two new problems emerge: selection and visualization. Users must determine which concepts are most relevant to a specific query and then efficiently browse thousands of results across multiple semantic dimensions, such as time, visual similarity, and concept relevance.

Methodology: The Semantic Pathfinder

The MediaMill system uses a "semantic pathfinder" architecture that breaks down video indexing into several sophisticated stages, from raw feature extraction to high-level concept detection.

The MediaMill semantic pathfinder architecture, illustrating the flow from visual feature extraction to combined analysis.

Multi-Level Visual Analysis

Instead of relying on a single feature type, the system extracts data at three distinct levels:

  • Regular Grids: Using "Wiccest" features and Gabor filters to capture texture and color statistics.
  • Interest Points: Utilizing SIFT and color descriptors clustered into a "visual words" codebook.
  • Segmented Blobs: Dividing images into meaningful objects or regions before analysis.

Fusion and Interaction

To combine results from different detectors, the team employed a late-fusion strategy using the geometric mean. This method proved more robust than simple multiplication as it gracefully handles cases where a specific detector may lack data for a given shot.

To address the visualization challenge, the team introduced the ForkBrowser. This novel UI visualizes results in a fork shape: the "stem" shows the history of viewed shots, while the "tines" represent different search threads (time, visual similarity, or query results).

The MediaMill Semantic Video Search Engine interface and the ForkBrowser, which visualizes search threads in a multi-tine structure.

Key Insights and Results

  • Lexicon Scale: The team successfully scaled their system to 572 concepts, a significant jump from the 491 used the previous year.
  • Data Volume Over Recency: In several instances, detectors trained on the older TRECVID 2005 dataset outperformed those trained on the 2007 set. For example, having 428 examples of an "airplane" from 2005 was more effective than having only 29 examples from 2007, despite the domain shift.
  • Efficiency in Interaction: The ForkBrowser achieved high performance while requiring significantly fewer user moves than previous interfaces, suggesting that multi-threaded visualization is key to scaling interactive search.
  • Benchmark Performance: The team ranked 3rd and 4th overall in the TRECVID 2007 interactive search benchmark, achieving top-3 average precision results for 8 out of 24 search topics.

Conclusion

The MediaMill 2007 entry demonstrates that scaling concept lexicons and improving interactive visualization can significantly enhance video retrieval. While limitations remain in temporal feature extraction and speech recognition quality, the success of the ForkBrowser and the robust performance of the semantic pathfinder architecture provide a strong foundation for future cross-domain video search research.

Summary updated

BibTeX
@inproceedings{snoek2007mediamill,
  author = {Cees G. M. Snoek and Ivo Everts and Jan C. van Gemert and Jan-Mark Geusebroek and Bouke Huurnink and Dennis Koelma and M. van Liempt and Ork de Rooij and Koen E. A. van de Sande and Arnold W. M. Smeulders and Jasper R. R. Uijlings and Marcel Worring},
  title = {{The MediaMill TRECVID 2007 Semantic Video Search Engine}},
  booktitle = {TRECVID},
  year = {2007},
  url = {https://openreview.net/forum?id=4nfCsamHiG},
}
Figure