TL;DR
Training natural language video localization models typically demands costly, timestamp-level annotations. While zero-shot approaches attempt to bypass this by generating synthetic "pseudo-queries," these generated descriptions often lack grounding in reality. CORONET bridges this gap by infusing external commonsense knowledge from ConceptNet into both video and query representations, substantially outperforming existing zero-shot and weakly supervised methods.
What Problem Does This Paper Address?
Natural Language Video Localization (NLVL) requires a model to locate the exact temporal start and end timestamps in an untrimmed video that match a user's textual query (e.g., "the man walks to the kitchen and cooks").
Training these systems is notoriously expensive because human annotators must meticulously watch hours of video to mark precise start and end boundaries. Recently, zero-shot NLVL emerged to train models solely on raw video. Existing zero-shot frameworks dynamically segment videos and generate synthetic "pseudo-queries." However, current approaches hit two major bottlenecks:
- Noisy query generation: Detecting objects with vision models and predicting actions using ungrounded language models introduces hallucinations and spurious verb-noun correlations.
- Language-free shortcuts: Bypassing pseudo-queries altogether using pretrained vision-language representations (like CLIP) discards rich relational and situational context.
The authors ask: Can structured commonsense knowledge provide the missing connective tissue to ground pseudo-queries into raw video features without human supervision?
Methods
The authors introduce CORONET (CommOnsense zeRo-shOt laNguage vidEo localizaTion), which integrates external knowledge subgraphs into zero-shot temporal regression:
[Raw Video] ──> Dynamic Moment Proposal (f_span) ──> Detected Objects (Top-k)
│
ConceptNet Subgraph
│
Video Features ──┐ ▼
├──> [Commonsense Enhancement Module] <── GCN Encoder
Query Features ──┘ (Cross-Attention via Concepts)
│
Cross-Modal Interaction (QVA & VQA)
│
Temporal Regression (Boundary Output)
- Dynamic Pseudo-Supervision: Moment proposals are generated via clustering visual frame features and concatenating temporal indices. Pertinent objects (nouns) within each segment are extracted using an off-the-shelf object detector.
- Video-Conditioned Commonsense Subgraphs: Instead of predicting ungrounded verbs with an external language model, CORONET samples the most frequent detected objects as seed concepts $C$ and builds a subgraph from ConceptNet. It explicitly filters for actionable relation types (e.g., spatial: LocatedNear; temporal: HasSubevent; causal: Causes), deliberately ignoring vague connections like RelatedTo.
- Concept Encoder: A Graph Convolutional Network (GCN) encodes topological and semantic information across concept nodes.
- Commonsense Enhancement Module (CEM): Separate cross-attention mechanisms enrich both the encoded video frames (from an I3D backbone + GRU) and query tokens (via a Bi-GRU) with guidance from the encoded commonsense concept embeddings.
- Cross-Modal Attention & Temporal Regression: Features undergo dual attention—Query-guided Video Attention (QVA) and Video-guided Query Attention (VQA)—using Attention Dynamic Filters before a temporal MLP predicts boundaries guided by a smooth $\ell_1$ regression loss and temporal attention loss. At test time, natural queries are simplified to noun tokens via a standard part-of-speech (POS) tagger.
Datasets
The authors evaluate solely using video data during training, holding back textual queries and temporal annotations strictly for evaluation across two standard NLVL benchmarks:
- Charades-STA: Indoor, action-dense videos featuring everyday home activities.
- ActivityNet-Captions: Open-domain, diverse, untrimmed videos covering hundreds of complex human activities.
Key Insights and Ideas
- Temporal and spatial relations are the MVPs: Ablations reveal that incorporating temporal relations (HasFirstSubevent, HasPrerequisite) provides significantly higher performance gains than spatial relations alone. Combining both spatial and temporal commonsense produces the steepest improvements.
- More concepts aren't always better: Scaling the concept graph from 250 to 300 nodes improves average localization coverage (mIoU) but can dilute fine-grained temporal precision ([email protected]), illustrating an inherent trade-off between broad domain generalization and boundary precision.
- Enabling co-reference resolution: Qualitative evaluations show CORONET resolves ambiguities like "same person" across multi-step actions where other baselines falter, because commonsense structures guide sequential event comprehension.
Key Results
CORONET surpasses existing zero-shot methods and several fully/weakly supervised baselines across temporal Intersection-over-Union thresholds (R@$k$, where $k \in \{0.3, 0.5, 0.7\}$) and mean IoU (mIoU):
- Charades-STA:
- Achieves 17.93% at [email protected] compared to 13.57% for the prior zero-shot baseline PSVL (a 32.1% relative improvement) and beats language-free LFVL (16.95%).
- Matches or exceeds the supervised CTRL baseline (which scored 21.42% at [email protected] and 7.15% at [email protected]).
- ActivityNet-Captions:
- Delivers 46.05% at [email protected] and 28.19% at [email protected], outperforming both PSVL (43.03% / 25.14%) and LFVL (43.34% / 25.17%).
Limitations and Future Work
- Over-reliance on nouns at test time: During inference, CORONET filters queries down to detected part-of-speech nouns. While this side-steps noisy verb hallucinations, it inherently discards complex adverbial or syntactic cues in intricate queries.
- Static knowledge bases: The framework relies on ConceptNet subgraphs. Concepts outside the predefined vocabulary or rare domain-specific actions may be absent.
- Dynamic graph scaling: The current subgraph structure requires manual relation filtering; future work could explore dynamic, end-to-end knowledge graph retrieval and pruning tailored per video instance.
Summary updated