Introduction
Natural Language Video Localization (NLVL) involves identifying the start and end timestamps of specific events in a video based on a natural language query. While supervised methods have achieved significant success, they rely on expensive human-annotated video-query pairs. Zero-shot approaches aim to eliminate this dependency by training on raw video data. However, these methods often struggle with the "semantic gap"—the disconnect between low-level visual features and the abstract concepts described in text queries.
Presented at AAAI 2024, CORONET (CommOnsense zeRo-shOt laNguage vidEo localizaTion) addresses this challenge by integrating external commonsense knowledge from ConceptNet into the localization pipeline.
The Semantic Gap in Zero-Shot NLVL
In zero-shot settings, models typically generate "pseudo-queries" using object detectors (for nouns) and language models (for verbs). These machine-generated queries are often noisy and lack the contextual depth of human language. For instance, a model might detect a "person" and a "pan" but fail to infer the broader context of "cooking" without explicit guidance. CORONET bridges this gap by leveraging a knowledge graph to provide the missing relational context.
The CORONET Framework
The CORONET architecture consists of several integrated modules designed to enhance both visual and textual representations with commonsense knowledge.
1. Pseudo-Query Generation
The process begins with a dynamic proposal module that extracts video segments. An off-the-shelf object detector identifies primary objects within these segments to form the basis of a training query.
2. Commonsense Enhancement Module (CEM)
The CEM is the core innovation of CORONET. It extracts a relevant subgraph from ConceptNet based on the detected objects. This subgraph is then encoded using a Graph Convolutional Network (GCN), allowing the model to learn semantic relationships (e.g., a "pan" is "UsedFor" "cooking").
3. Pre-fusion Enhancement
Unlike traditional methods that fuse modalities early, CORONET enhances video and query features separately before interaction. Using cross-attention mechanisms—Query-guided Video Attention (QVA) and Video-guided Query Attention (VQA)—the model aligns visual features with commonsense-enriched textual concepts.
4. Temporal Regression
The final enriched features are processed through a Multi-Layer Perceptron (MLP) to predict the precise start ($t_s$) and end ($t_e$) timestamps of the event.
Experimental Evaluation
CORONET was evaluated on two major benchmarks: Charades-STA (daily indoor activities) and ActivityNet-Captions (diverse open-domain events). The model was trained exclusively on raw videos, with human labels reserved for testing.
Key Findings
- Superior Performance: CORONET established new state-of-the-art results for zero-shot localization. On Charades-STA, it outperformed the previous leading method (PSVL) by 32.13% in high-precision ([email protected]) scenarios.
- Outperforming Weak Supervision: Remarkably, CORONET surpassed several weakly supervised baselines that had access to ground-truth text queries during training.
- Importance of Temporal Relations: Ablation studies revealed that temporal relations (e.g., "HasSubevent") in the knowledge graph were more impactful than spatial relations (e.g., "AtLocation") for the localization task.
- Architecture Insights: Simple weighted GCNs outperformed more complex relational variants, and GRU-based encoders proved more effective than Transformers in this specific zero-shot context.
Conclusion
CORONET demonstrates that external commonsense knowledge is a powerful tool for bridging the semantic gap in zero-shot video understanding. By enriching noisy pseudo-queries with structured relational data, the framework achieves high-precision localization without the need for costly human annotations. Future research may focus on refining knowledge graph filtering and exploring "visual commonsense" to further improve action grounding.
Summary updated