IVA-NLPR-IA-CAS TRECVID 2009: High LevelFeatures Extraction

Jinqiao Wang, Si Liu, Chao Liang, Hanqing Lu

Abstract

This paper details the participation of the Chinese Academy of Sciences (IVA-NLPR-IA-CAS) in the TRECVID 2009 High-Level Feature (HLF) extraction task. The research explores three distinct strategies to mitigate the extreme data imbalance found in video concept detection: a baseline system using negative sample division, a visual concept network for training set expansion, and web data transfer from Flickr. The findings highlight the significant challenges posed by the "domain gap" when using external web data and the "semantic gap" when leveraging related concepts.

Introduction: The Challenge of Data Imbalance

In video retrieval, High-Level Feature (HLF) extraction is the task of automatically detecting semantic concepts—such as "airplane flying," "classroom," or "singing"—within video clips. The primary hurdle in this domain is extreme data imbalance. In massive datasets like TRECVID, positive examples of a specific concept are incredibly rare compared to the vast number of negative examples.

The positive-to-negative sample ratio for various concepts in the TRECVID dataset, highlighting the extreme data imbalance that makes training robust classifiers difficult.

Standard machine learning models, such as Support Vector Machines (SVMs), often struggle with such skewed datasets, frequently defaulting to negative predictions to maintain nominal accuracy. The authors set out to find better ways to balance the scales, testing whether they could "borrow" positive samples from related concepts or external web sources like Flickr.

Methodology

The IVA-NLPR-IA-CAS system utilizes a combination of global and local visual features:

  • Feature Extraction: Researchers used "Grid Features" (capturing texture, color moments, and edges) and local SIFT descriptors. These local features were aggregated into a 500-dimension "Bag-of-Words" (BoW) representation using a soft-weighting scheme to improve robustness.
  • Experimental Runs:
    1. Baseline (Run 1): Addressed imbalance via "negative sample division," splitting the negative data into smaller chunks to train multiple SVMs, then fusing the results.
  • Visual Concept Network (Run 2): Used Affinity Propagation (AP) clustering to find concepts that frequently co-occur (e.g., "boat" and "harbor"). The positive training set for a target concept was expanded by including samples from its semantic "neighbors."
    1. Web Data Transfer (Run 3): For "sparse" concepts with very few examples, the team crawled the top 500 images from Flickr using expanded keywords (e.g., searching for "student" and "teacher" for the concept "classroom") to bolster the training set.

Key Results and Discussion

The performance of the three strategies varied significantly, illustrating the difficulty of naive data expansion.

Comparison of inferred average precision across the three experimental runs (Baseline, Concept Network, and Web Transfer) against median and best performance per feature.

The "Noisy Neighbor" Problem

While grouping related concepts like "bus" and "street" is semantically intuitive, it proved to be a double-edged sword. Including "street" images helped the model find more buses (higher recall), but it also caused the model to mistakenly flag streets without buses as "bus" shots, leading to a significant loss in precision.

The Domain Gap in Web Data

One of the most striking findings was the failure of web-crawled data. Despite the high quality of Flickr images, they did not translate well to video frames. The Average Precision dropped from the baseline's 0.162 to 0.055 in Run 3. The authors identified three specific reasons for this performance drop:

  1. Resolution: Flickr images are high-resolution; TRECVID video frames are often blurry or low-resolution.
  2. Object Scale: Flickr photos usually feature large, centered subjects, whereas objects in TRECVID videos are often small, distant, or partially obscured.
  3. Background Complexity: Flickr backgrounds are often clean or aesthetically pleasing, while real-world video contains "messy" and complicated backgrounds.

Conclusion

The IVA-NLPR-IA-CAS experiments demonstrate that simply adding more data is not helpful if that data introduces semantic noise or comes from a different domain. The "semantic gap" between concepts and the "domain gap" between web images and video frames remain major obstacles. Future work will focus on more sophisticated domain adaptation algorithms that can account for differences in resolution, scale, and background complexity between disparate data sources.

Summary updated

Abstract

In this report, we present overview and comparative analysis of our HLF detection system. Our baseline method utilizes global and local feature and reaches MAP 0.162. As many concepts in trecvid competition are relevant, we can mine relations between them to help train robust classifier. The last run is to utilize images from web data to enhance high level feature training. Three runs are submitted as following:  A_IVA_NLPR_IA_CAS1_1: baseline appraoch– average fusion of 4 SVM classification results for each concept using various feature representation choices.  A_IVA_NLPR_IA_CAS2_2: visual concept network based approach.  C_IVA_NLPR_IA_CAS3_3: web data transfer based approach.
BibTeX
@inproceedings{wang2009iva,
  author = {Jinqiao Wang and Si Liu and Chao Liang and Hanqing Lu},
  title = {{IVA-NLPR-IA-CAS TRECVID 2009: High LevelFeatures Extraction}},
  booktitle = {TRECVID},
  year = {2009},
  url = {https://openreview.net/forum?id=r6DYwBNunK8},
}
Figure