MESH participation to TRECVID2008 HLFE

Javier Molina, José María Martínez Sanchez, Vasileios Mezaris, Georgios Th. Papadopoulos, Spiros Nikolopoulos, Ioannis Kompatsiaris, Anastasios Dimou, Paulo Villegas, J. Rodríguez-Benito, Eduard Bru, Tomasz Adamek, Giorgos Tolias, Evaggelos Spyrou, Natasa Sofou, Petros Kapsalas, Yannis AvrithisTRECVID 2008

Abstract

The MESH consortium participated in the TRECVID 2008 High Level Feature Extraction (HLFE) task, deploying an automated pipeline for semantic concept detection in video. The approach explored various configurations, including MPEG-7 baselines, SIFT Bag-of-Words, and multi-stage classification strategies. Results highlighted the effectiveness of late fusion (combining SIFT and MPEG-7) but also revealed how a critical implementation bug in the baseline training phase can propagate through dependent models, underscoring the importance of robust validation in automated systems.

Introduction

High Level Feature Extraction (HLFE) involves the automatic detection of semantic concepts—such as "Cityscape," "Two People," or "Nighttime"—within video shots. For TRECVID 2008, the MESH consortium adapted its news-tuned visual analysis infrastructure to a broader concept set. A primary goal was to develop a "blind" training system capable of automatically selecting optimal models and feature configurations without human intervention.

Methodology

The MESH team submitted six distinct system configurations, or "runs," primarily based on a Support Vector Machine (SVM) architecture. These runs explored different feature sets and classification strategies:

  1. MPEG-7 Baseline (Run 4): Utilized global visual descriptors (e.g., color layout, edge histogram) with an RBF-kernel SVM.
  2. SIFT Bag of Words (Run 5): Employed a 10,000-word visual vocabulary derived from random SIFT keypoints.
  3. Visual Words & Part-Based Detection (Run 3): Combined Haar/Harris features for object parts with custom color-segmentation for scene concepts.
  4. 2-Stage Classification (Run 6): Aimed to capture concept correlations by feeding baseline predictions into a second-stage SVM.
  5. Extended 2-Stage (Run 1): Augmented the 2-stage setup with HOG descriptors and motion activity features.
  6. Combined SIFT & MPEG-7 (Run 2): A late-fusion approach merging global MPEG-7 and local SIFT features.
Relationship and hierarchy of the six submitted runs.

The system relied on TRECVID 2008 development data and collaborative annotations for training. Additionally, the PASCAL VOC2006 dataset was used to train part-based detectors for specific concepts in Run 3.

Results and Discussion

The experimental results provided a stark contrast between the potential of the architecture and the impact of implementation errors. Run 2, the late-fusion model, was the most successful, performing near the median of all TRECVID participants.

Summary of performance metrics (Total true shots and Mean inferAP) across all six runs.

A significant finding was the "cascading bug" effect. An error in calculating Mean Average Precision (MAP) during the baseline training phase—specifically the omission of false negatives—led to severe overfitting. Since Runs 1, 3, and 6 were dependent on this baseline, their performance was significantly degraded. Despite this, the models maintained decent precision at the very top of ranked lists (e.g., 20% precision at 5 shots), suggesting they successfully identified obvious concept instances but failed on more complex, "long-tail" examples.

Conclusion

The MESH participation in TRECVID 2008 serves as both a demonstration of advanced fusion techniques and a cautionary tale regarding pipeline integrity. While the late-fusion approach (Run 2) proved resilient, the failure of dependent models due to a baseline bug highlights the necessity of rigorous cross-validation and error-checking in automated machine learning workflows. Future work will focus on correcting these optimization bugs, refining the SIFT Bag-of-Words approach, and developing better sampling strategies for highly unbalanced datasets.

Summary updated

BibTeX
@inproceedings{molina2008mesh,
  author = {Javier Molina and José María Martínez Sanchez and Vasileios Mezaris and Georgios Th. Papadopoulos and Spiros Nikolopoulos and Ioannis Kompatsiaris and Anastasios Dimou and Paulo Villegas and J. Rodríguez-Benito and Eduard Bru and Tomasz Adamek and Giorgos Tolias and Evaggelos Spyrou and Natasa Sofou and Petros Kapsalas and Yannis Avrithis},
  title = {{MESH participation to TRECVID2008 HLFE}},
  booktitle = {TRECVID},
  year = {2008},
}
Figure