Introduction
The "semantic gap"—the disconnect between low-level digital signals (pixels) and high-level human concepts—remains a primary obstacle in content-based video retrieval. In their 2007 TRECVID entry, Fudan University addressed this challenge by leveraging Multi-Task Learning (MTL) and Bayesian Networks to model concept relationships and fuse multi-modal data. Their approach specifically targeted hurdles such as data scarcity, class imbalance, and the effective integration of text, image, and motion information.
Methodology
The research team implemented two primary workflows: High-Level Feature Extraction (HLFE) and Interactive Search.
High-Level Feature Extraction (HLFE)
The HLFE pipeline focuses on extracting meaningful concepts from video shots by combining low-level features with advanced learning models. Due to the high volume of monochrome clips in the 2007 dataset, the team prioritized texture-based features, including Gabor filters and Gray Level Co-occurrence Matrices (GLCM), over color descriptors.
To combat data scarcity, the team employed Multi-Task Learning (MTL). By clustering related tasks (e.g., grouping "Bus," "Car," and "Truck"), the model could learn shared characteristics across categories, which is particularly beneficial for rare concepts. Furthermore, Bayesian Networks were utilized to model dependencies between concepts. This allowed the system to refine predictions based on logical hierarchies; for example, high confidence in a "Face" detection would increase the probability of a "Person" being present.
Interactive Search
The search system utilized a multi-modal approach, querying four distinct models:
- Text: ASR and MT transcripts.
- Image: Global MPEG-7 visual features.
- Visual Concepts: An ontology mapping queries to detected HLFE features.
- Camera Motion: Detection of tilts and zooms directly from MPEG motion vectors.
Two fusion strategies were evaluated: AP-Based Borda Count (BC), which weights results by precision, and Mixed Group Ranks (MGR), which incorporates user feedback.
Key Results
The experimental results demonstrated that the integration of Bayesian Networks consistently improved performance across all feature extraction runs.
As shown in Figure 2, runs incorporating Bayesian Network fusion (such as A_FD_SVM_BN_2) generally outperformed their counterparts. While standard SVMs remained effective for data-rich concepts, MTL proved advantageous for the "long tail" of rare categories like "Animal" or "Charts." In the search task, the FUDAN_P run (using BC fusion) achieved the highest precision, while the FUDAN_R run (using MGR fusion) achieved the highest recall, confirming a clear trade-off between the two strategies.
Conclusion
Fudan University's 2007 TRECVID participation highlighted the power of modeling concept dependencies and adaptive feature selection. While the system struggled with complex contextual actions, such as "people walking upstairs," the use of Bayesian Networks and MTL provided a robust framework for narrowing the semantic gap in large-scale video datasets.
Summary updated