Abstract
The user base of short video apps has experienced unprecedented growth in recent years, resulting in a significant demand for video content analysis. In particular, text-video retrieval, which aims to find the top matching videos given text descriptions from a vast video corpus, is an essential function, the primary challenge of which is to bridge the modality gap. Nevertheless, most existing approaches treat texts merely as discrete tokens and neglect their syntax structures. Moreover, the abundant spatial and temporal clues in videos are often underutilized due to the lack of interaction with text. To address these issues, we argue that using texts as guidance to focus on relevant temporal frames and spatial regions within videos is beneficial. In this paper, we propose a novel Syntax-Hierarchy-Enhanced text-video retrieval method (SHE-Net) that exploits the inherent semantic and syntax hierarchy of texts to bridge the modality gap from two perspectives. First, to facilitate a more fine-grained integration of visual content, we employ the text syntax hierarchy, which reveals the grammatical structure of text descriptions, to guide the visual representations. Second, to further enhance the multi-modal interaction and alignment, we also utilize the syntax hierarchy to guide the similarity calculation. We evaluated our method on four public text-video retrieval datasets of MSR-VTT, MSVD, DiDeMo, and ActivityNet. The experimental results and ablation studies confirm the advantages of our proposed method.
Introduction
Text-video retrieval—the task of identifying the most relevant video for a given natural language query—is central to modern content discovery. However, a significant "modality gap" persists between text and video. Text is a highly structured, human-generated medium governed by grammar, whereas video is a high-dimensional, often redundant sequence of visual signals.
Existing retrieval models typically treat text as a flat sequence of tokens and videos as a sequence of frames, failing to exploit the hierarchical relationships between linguistic components and visual entities. For example, a verb in a sentence often corresponds to a specific temporal window of action, while a noun corresponds to a physical object in a spatial region. SHE-Net addresses this by using the grammatical structure of the text to guide the model's attention across both temporal and spatial dimensions of the video.
Methodology
SHE-Net employs a hierarchical approach that mirrors linguistic structure in the visual domain through three primary components:
1. Text and Video Syntax Hierarchies
The model first parses the input text into a four-level hierarchy: Sentence $\rightarrow$ Verbs $\rightarrow$ Nouns $\rightarrow$ Adjectives. This linguistic structure then guides the construction of a corresponding three-tier visual hierarchy: Video Content $\rightarrow$ Frames $\rightarrow$ Patches.
- Temporal Guidance: Verbs in the text hierarchy act as anchors to select and aggregate relevant temporal frames.
- Spatial Guidance: Nouns are mapped to specific spatial patches within those frames, allowing the model to focus on relevant entities like "horse" or "shirt" while ignoring background noise. Adjectives are further utilized to refine noun representations (e.g., distinguishing a "green shirt" from a "red shirt").
2. Multi-Granularity Alignment
Instead of a single global similarity score, SHE-Net computes similarities at multiple granularities: Global, Verb-Frame, and Noun-Patch. This ensures that both high-level semantics and fine-grained details are captured.
3. Syntax-Enhanced Similarity Calculation
The final retrieval score is a weighted sum of these hierarchical similarities. The weights are dynamically learned based on the importance of specific nodes within the syntax tree, ensuring that critical semantic elements contribute more to the final ranking.
Experimental Results
The model was rigorously evaluated on four benchmark datasets: MSR-VTT, MSVD, DiDeMo, and ActivityNet.
Quantitative Performance
SHE-Net consistently outperformed state-of-the-art baselines:
- MSR-VTT: Achieved an Rsum of 212.3 (with DSL post-processing), a significant improvement over previous methods like CLIP4Clip.
- MSVD: Reached an R@1 of 47.6%, demonstrating superior retrieval accuracy.
- Efficiency: The hierarchical processing adds minimal overhead, with only a 15% increase in training time and a 5% increase in inference time compared to non-hierarchical baselines like TS2-Net.
Qualitative Analysis
Qualitative results demonstrate the model's ability to accurately localize actions. As shown in the figure below, the model assigns higher similarity scores to frames where the action described by the verb (e.g., "hits," "climbing," "singing") is most prominent.
Conclusion
SHE-Net demonstrates that grammatical structure is a powerful prior for bridging the text-video modality gap. By treating verbs as temporal anchors and nouns as spatial anchors, the model achieves more precise alignment and superior retrieval performance. Future work will explore more robust parsing for informal queries and adaptive selection of visual granularities.
Summary updated