Zhejiang University at TRECVID 2006

Yanan Liu, Fei Wu, Yueting Zhuang, Shengyi ZhouTRECVID 2006 · OpenReview

Abstract

In the TRECVID 2006 challenge, researchers from Zhejiang University addressed the "semantic gap" in video retrieval—the disconnect between low-level digital features and high-level human concepts. By moving beyond traditional early and late fusion techniques, the team proposed a framework leveraging SimFusion for cross-modal similarity propagation and Locality Preserving Projections (LPP) for efficient dimensionality reduction. Their approach treats video, audio, and text as a unified, temporally linked entity, specifically targeting the Temporal-Sequenced Associated Co-occurrence (TSAC) inherent in multimedia data.

Introduction

The "semantic gap" remains a primary obstacle in multimedia retrieval. While computers excel at processing pixels and frequencies, translating these into concepts like "meeting" or "outdoor scene" is complex. Most existing systems rely on either early fusion (merging raw data at the start) or late fusion (averaging results from independent modalities). However, these methods often ignore Temporal-Sequenced Associated Co-occurrence (TSAC): the fact that visual, auditory, and textual components in a video clip occur simultaneously and describe the same event. This research explores a multi-modal pipeline designed to exploit these cross-modal relationships to improve semantic detection and search.

Methodology

The proposed system utilizes a multi-stage pipeline to process video shots into a searchable semantic space.

1. Multi-Modal Feature Extraction

To capture the breadth of the video data, the team extracted features across three primary modalities:

  • Visual: Keyframes were analyzed for color histograms, textures, and edge distributions.
  • Audio: Statistical measures, including the mean and variance of short-time audio frames, were calculated.
  • Text: Automatic Speech Recognition (ASR) transcripts were processed using Latent Semantic Analysis (LSA) to identify underlying thematic structures rather than relying solely on keyword matching.

2. SimFusion: The Unified Similarity Matrix

The core novelty of the system is the use of SimFusion. The researchers constructed a Unified Relationship Matrix (URM) that maps the relationships between every shot across all modalities. By iteratively calculating a Unified Similarity Matrix (USM), the system allows a strong similarity in one modality (e.g., visual similarity) to reinforce or "propagate" similarity in another (e.g., audio). This ensures that the multi-modal nature of the video is treated as a cohesive feature set rather than isolated variables.

3. Dimensionality Reduction via LPP

To manage the "curse of dimensionality" inherent in multi-modal data, the team employed Locality Preserving Projections (LPP). Unlike standard linear methods like Principal Component Analysis (PCA), LPP is designed to discover the intrinsic non-linear structure of the data manifold. As a linear approximation of a non-linear manifold, LPP is computationally efficient and allows the system to easily map new, unseen video shots into the established semantic space.

For high-level feature detection, Support Vector Machines (SVM) were trained on the reduced semantic space. For the search task, the team implemented a relevance feedback loop. This allowed the system to refine search coordinates in real-time based on user interactions, learning from positive and negative examples to improve retrieval accuracy.

Experimental Setup and Results

The experiments were conducted using the TRECVID 2006 dataset. The team focused specifically on the English news data, excluding the Chinese and Arabic portions of the collection.

High-Level Feature Extraction

Despite using only a subset of the available training data, the system's performance was competitive, generally hovering around the median level across most concepts. Notably, the system exceeded median performance for specific concepts such as #26 and #36, suggesting that the SimFusion approach is particularly effective for certain semantic categories.

The interactive search results were slightly below the median. The authors attributed this to their status as first-time participants, noting a need for a more nuanced understanding of query topics and a more refined relevance feedback mechanism to guide the user experience.

Discussion and Future Work

The primary limitation of this study was the restricted data scope; by omitting non-English news sources, the model lacked exposure to a significant portion of the TRECVID training set.

Future research will focus on developing concept-specific processes. Rather than applying a uniform algorithm to all queries, the team aims to tailor fusion and detection parameters to specific topics. For instance, a "weather" concept might prioritize textual ASR data, while a "sports" concept might rely more heavily on specific audio-visual patterns. By refining these weights, the researchers hope to further bridge the semantic gap in complex video environments.

Summary updated

BibTeX
@inproceedings{liu2006zhejiang,
  author = {Yanan Liu and Fei Wu and Yueting Zhuang and Shengyi Zhou},
  title = {{Zhejiang University at TRECVID 2006}},
  booktitle = {TRECVID},
  year = {2006},
  url = {https://openreview.net/forum?id=k-hofb2Ms3-},
}
Figure