Abstract
Video Paragraph Grounding (VPG) involves the simultaneous localization of multiple events described in a multi-sentence paragraph within a video. Unlike standard Video Sentence Grounding, VPG requires maintaining temporal order and capturing both fine-grained actions and global context. We present the Hierarchical Semantic Correspondence Network (HSCNet), a novel framework that aligns video and text across three levels: words, sentences, and paragraphs. By employing a bottom-up hierarchical encoder and a top-down progressive decoder, HSCNet effectively leverages multi-level semantic relationships. Experimental results on ActivityNet-Captions and TACoS datasets demonstrate that HSCNet significantly outperforms existing state-of-the-art methods.
Introduction
Video Sentence Grounding (VSG) has seen significant progress, yet the more complex task of Video Paragraph Grounding (VPG) remains challenging. VPG requires a model to interpret a paragraph and localize every described event while preserving their temporal sequence. Existing approaches typically operate at the sentence level, which often overlooks two vital sources of information: fine-grained word-level details (e.g., specific verbs like "grabs") and global paragraph-level context.
To address these limitations, we propose modeling the hierarchical relationship between words, sentences, and paragraphs. As shown in Figure 1, fine-grained details provide precise cues for temporal boundaries, while the global context helps filter irrelevant background noise and resolve ambiguities across the video timeline.
Methodology: HSCNet
The Hierarchical Semantic Correspondence Network (HSCNet) utilizes a "sandwich" architecture consisting of a hierarchical multi-modal encoder and a hierarchical progressive decoder.
Hierarchical Multi-modal Encoder (Bottom-Up)
The encoder builds semantic representations starting from the most granular level:
- Word Level: Aligns video clips with verbs and nouns to capture specific action dynamics.
- Sentence Level: Aggregates word-level information to focus on event-centric relations.
- Paragraph Level: Captures the holistic "story" of the video, connecting disparate events.
At each level, Multi-modal Self-Attention is employed to facilitate interaction between video and text features, identifying shared semantic grounding.
Hierarchical Progressive Decoder (Top-Down)
The decoder refines temporal boundaries in a top-down fashion. It first predicts the boundary for the entire paragraph, which then serves as a spatial-temporal filter for sentence-level grounding. Finally, word-level grounding is performed. This progressive refinement ensures that fine-grained predictions are consistent with the broader narrative context established at higher levels of the hierarchy.
Experiments and Results
HSCNet was evaluated on two major benchmarks: ActivityNet-Captions and TACoS.
- ActivityNet-Captions: HSCNet achieved a mean Intersection over Union (mIoU) of 59.71%, representing a nearly 4% improvement over the previous state-of-the-art (SVPTR).
- TACoS: On this challenging dataset characterized by long videos and dense paragraphs, HSCNet reached 40.61% mIoU, outperforming previous methods by over 9%.
Key Insights
Ablation studies highlighted the importance of the hierarchical structure. A version of the model restricted to sentence-level data performed significantly worse (32.66% mIoU on TACoS) than the full three-level version. Furthermore, word-level alignment proved to be highly impactful; even without explicit word-level timestamps, the model learned effective word-video correspondences through a "union and subset" loss constraint that tied word predictions to sentence boundaries.
Conclusion
HSCNet demonstrates the efficacy of hierarchical modeling for Video Paragraph Grounding. By aligning text and video across multiple semantic levels, the model captures both the nuances of individual actions and the overall narrative structure. While challenges like visual occlusion remain, this hierarchical approach sets a new standard for complex temporal grounding tasks.
Summary updated