VITALAS at TRECVID-2008

Christos Diou, Christos Papachristou, Panagiotis Panagiotopoulos, Anastasios Delopoulos, George Stephanopoulos, Nikos Dimitriou, Henning Rode, Arjen P. de Vries, Theodora Tsikrika, Robin Aly

Introduction

The VITALAS consortium's participation in the TRECVID 2008 benchmark focused on two primary challenges in video retrieval: High-Level Feature Extraction (HLFE) and Search. The core challenge is the "semantic gap"—the disconnect between low-level visual data (edges, colors, textures) and high-level human concepts (e.g., "a person driving a car"). The team explored bridging this gap by moving beyond raw pixels to a framework where concepts themselves become the building blocks for discovery.

Methodology

The authors employed a multi-layered approach that combined standard computer vision with novel fusion techniques. A central component was Concept Score Fusion. Instead of relying solely on low-level features, the system used the output scores of 89 different concept detectors (from the LSCOM ontology) as input features for a final classifier. This approach leverages the correlation between concepts; for instance, the presence of "water" and "sky" increases the probability of a "boat."

The VITALAS feature extraction and classification pipeline, illustrating the fusion of low-level features with concept scores and subsequent feature selection.

To manage the high dimensionality of the feature space (over 2,000 features), the team used mutual information-based algorithms for Feature Selection, pruning the space down to the most informative few hundred dimensions. For search tasks, video shots were converted into "artificial text documents" where textual descriptions of detected concepts were weighted by their probability.

Datasets and Evaluation

The primary development and training set was TRECVID 2005, utilizing the Large-Scale Concept Ontology for Multimedia (LSCOM) annotations. Evaluation was conducted on the TRECVID 2008 test set, incorporating collaborative annotations from the LIG and LIRIS groups.

Key Results and Insights

The experimental results highlighted the effectiveness of concept-based features and rigorous feature selection.

  • HLFE Performance: "Run 2," which integrated concept score fusion with feature selection, was the top-performing model. It achieved a Mean Inferred Average Precision (infAP) of 0.0338, outperforming configurations that relied on audio or complex clustering.
  • Search Success: The combined ASR-DEVTOP20 run, merging Automatic Speech Recognition (ASR) with the top 20 visual concepts, achieved the best search results with an infAP of 0.0238.
Performance results (Inferred Average Precision) for 20 high-level features across five experimental runs, showing Run 2 as the most effective overall.

A key insight was that features do not need to be logically related to be useful; a "Dog" detector might benefit from "Kitchen" detector outputs due to visual co-occurrence. Furthermore, reducing the feature space from over 800 to approximately 350 dimensions consistently improved performance, demonstrating that pruning noise is as critical as gathering data.

Limitations and Future Work

The study noted a significant performance drop when testing models trained on 2005 data on the 2008 dataset (Run 5), suggesting a degree of overfitting to specific filming styles. Additionally, simple k-means clustering did not yield significant gains, indicating that the feature space might be too noisy for basic unsupervised methods. While competitive for the time, the low absolute precision values underscore the ongoing difficulty of fully automated semantic video understanding.

Summary updated

Abstract

In this paper, we present our experiments in TRECVID 2008 about High-Level feature extraction task. This is the first year for our participation in TRECVID, our system adopts some popular approaches that other workgroups proposed before. We proposed 2 advanced low-level features NEW Gabor texture descriptor and the Compact-SIFT Codeword histogram. Our system applied well-known LIBSVM to train the SVM classifier for the basic classifier. In fusion step, some methods were employed such as the Voting, SVM-base, HCRF and Bootstrap Average AdaBoost(BAAB).
BibTeX
@inproceedings{diou2008vitalas,
  author = {Christos Diou and Christos Papachristou and Panagiotis Panagiotopoulos and Anastasios Delopoulos and George Stephanopoulos and Nikos Dimitriou and Henning Rode and Arjen P. de Vries and Theodora Tsikrika and Robin Aly},
  title = {{VITALAS at TRECVID-2008}},
  booktitle = {TRECVID},
  year = {2008},
  url = {https://openreview.net/forum?id=Ja3Su51DDOrr},
}
Figure