TL;DR
SHE-Net (Syntax-Hierarchy-Enhanced Network) improves text-video retrieval by using the grammatical structure of a sentence (verbs, nouns, and adjectives) to guide how a model "looks" at a video. By aligning verbs with specific temporal frames and nouns with specific spatial regions, the model achieves state-of-the-art performance across major benchmarks like MSR-VTT and MSVD.
What Problem Does This Paper Address?
Text-video retrieval—finding the most relevant video for a given text query—faces a massive "modality gap." Text is human-generated, concise, and highly structured by grammar. Video, conversely, is raw, high-dimensional, and full of redundant information.
Most existing models treat text as a simple sequence of tokens and videos as a sequence of frames, often ignoring the rich hierarchical relationship between them. For example, a "verb" in a sentence usually corresponds to an action occurring over a specific time window, while a "noun" corresponds to a physical entity located in a specific spatial region. Without leveraging this syntax, models often struggle to filter out irrelevant visual noise or capture fine-grained interactions.
Methods
The authors propose a hierarchical approach that mirrors the structure of language in the visual domain:
- Text Syntax Hierarchy: Using off-the-shelf tools (like spaCy), they parse the input text into a four-level hierarchy: Sentence $\rightarrow$ Verbs $\rightarrow$ Nouns $\rightarrow$ Adjectives.
- Video Syntax Hierarchy: Instead of looking at the whole video at once, the model builds a corresponding three-tier visual structure (Video Content $\rightarrow$ Frames $\rightarrow$ Patches) guided by the text:
- Temporal Guidance: Each verb in the text hierarchy is used to "retrieve" and aggregate a fixed number of relevant frames.
- Spatial Guidance: Each noun is mapped to specific image patches or regions within those frames.
- Multi-Granularity Alignment: Rather than calculating a single global similarity score, SHE-Net calculates similarities at multiple levels (Global, Verb-Frame, and Noun-Patch).
- Syntax-Enhanced Similarity: The final retrieval score is a weighted sum of these hierarchical similarities, where the weights themselves are learned based on the importance of specific words in the syntax tree.
Datasets
The model was evaluated on four standard public text-video retrieval benchmarks:
- MSR-VTT: 10,000 video clips with 200,000 captions.
- MSVD: 1,970 video clips with roughly 80,000 captions.
- DiDeMo: 10,000 videos from Flickr, focused on localized events.
- ActivityNet: Large-scale dataset with long videos and complex descriptions.
Key Insights and Ideas
- Verbs as Temporal Anchors: The "aha moment" here is explicitly treating verbs as temporal pointers. By using verb embeddings to select frames, the model effectively performs a soft form of temporal action localization.
- Nouns as Spatial Anchors: Similarly, nouns act as spatial pointers. This allows the model to ignore background clutter and focus on the entities (e.g., "horse," "shirt") mentioned in the text.
- Adjectives as Refiners: The authors found that adding a fourth layer for adjectives significantly improved the representation of nouns (e.g., distinguishing between a "red car" and a "blue car").
- Asymmetric Modalities: The paper acknowledges that text and video are inherently asymmetric. Text provides the instructions on how to compress the redundant video signal into "information-dense" features.
Key Results
SHE-Net consistently outperformed strong baselines like CLIP4Clip and TS2-Net:
- MSR-VTT: Achieved an Rsum of 212.3 (with DSL post-processing), a significant jump over previous state-of-the-art methods.
- MSVD: Reached an R@1 of 47.6%, outperforming the nearest competitor by 0.6%.
- Efficiency: Despite the hierarchical complexity, the model only adds about 15% to training time and 5% to inference time compared to the baseline TS2-Net, as many components (like the text parsing) can be pre-computed.
Limitations and Future Work
- Parser Dependency: The method relies on the accuracy of off-the-shelf syntax parsers. If the parser fails to identify the correct grammatical structure (especially in informal or ungrammatical user queries), the hierarchical alignment may suffer.
- Fixed Hyperparameters: The number of frames ($\lambda_{frame}$) and patches ($\lambda_{patch}$) to select are fixed hyperparameters. In reality, some actions take longer than others, and some entities are larger than others.
- Future Direction: The authors suggest that integrating more complex reasoning, such as causal relationships between actions in the hierarchy, could further enhance retrieval for long-form videos.
Summary updated