SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval

Xuzheng Yu, Chen Jiang, Xingning Dong, Tian Gan, Ming Yang, Qingpei Guo

TL;DR

SHE-Net (Syntax-Hierarchy-Enhanced Network) improves text-video retrieval by using the grammatical structure of a sentence (verbs, nouns, and adjectives) to guide how a model "looks" at a video. By aligning verbs with specific temporal frames and nouns with specific spatial regions, the model achieves state-of-the-art performance across major benchmarks like MSR-VTT and MSVD.

What Problem Does This Paper Address?

Text-video retrieval—finding the most relevant video for a given text query—faces a massive "modality gap." Text is human-generated, concise, and highly structured by grammar. Video, conversely, is raw, high-dimensional, and full of redundant information.

Most existing models treat text as a simple sequence of tokens and videos as a sequence of frames, often ignoring the rich hierarchical relationship between them. For example, a "verb" in a sentence usually corresponds to an action occurring over a specific time window, while a "noun" corresponds to a physical entity located in a specific spatial region. Without leveraging this syntax, models often struggle to filter out irrelevant visual noise or capture fine-grained interactions.

Methods

The authors propose a hierarchical approach that mirrors the structure of language in the visual domain:

  1. Text Syntax Hierarchy: Using off-the-shelf tools (like spaCy), they parse the input text into a four-level hierarchy: Sentence $\rightarrow$ Verbs $\rightarrow$ Nouns $\rightarrow$ Adjectives.
  2. Video Syntax Hierarchy: Instead of looking at the whole video at once, the model builds a corresponding three-tier visual structure (Video Content $\rightarrow$ Frames $\rightarrow$ Patches) guided by the text:
  3. Temporal Guidance: Each verb in the text hierarchy is used to "retrieve" and aggregate a fixed number of relevant frames.
  4. Spatial Guidance: Each noun is mapped to specific image patches or regions within those frames.
  5. Multi-Granularity Alignment: Rather than calculating a single global similarity score, SHE-Net calculates similarities at multiple levels (Global, Verb-Frame, and Noun-Patch).
  6. Syntax-Enhanced Similarity: The final retrieval score is a weighted sum of these hierarchical similarities, where the weights themselves are learned based on the importance of specific words in the syntax tree.

Datasets

The model was evaluated on four standard public text-video retrieval benchmarks:

  • MSR-VTT: 10,000 video clips with 200,000 captions.
  • MSVD: 1,970 video clips with roughly 80,000 captions.
  • DiDeMo: 10,000 videos from Flickr, focused on localized events.
  • ActivityNet: Large-scale dataset with long videos and complex descriptions.

Key Insights and Ideas

  • Verbs as Temporal Anchors: The "aha moment" here is explicitly treating verbs as temporal pointers. By using verb embeddings to select frames, the model effectively performs a soft form of temporal action localization.
  • Nouns as Spatial Anchors: Similarly, nouns act as spatial pointers. This allows the model to ignore background clutter and focus on the entities (e.g., "horse," "shirt") mentioned in the text.
  • Adjectives as Refiners: The authors found that adding a fourth layer for adjectives significantly improved the representation of nouns (e.g., distinguishing between a "red car" and a "blue car").
  • Asymmetric Modalities: The paper acknowledges that text and video are inherently asymmetric. Text provides the instructions on how to compress the redundant video signal into "information-dense" features.

Key Results

SHE-Net consistently outperformed strong baselines like CLIP4Clip and TS2-Net:

  • MSR-VTT: Achieved an Rsum of 212.3 (with DSL post-processing), a significant jump over previous state-of-the-art methods.
  • MSVD: Reached an R@1 of 47.6%, outperforming the nearest competitor by 0.6%.
  • Efficiency: Despite the hierarchical complexity, the model only adds about 15% to training time and 5% to inference time compared to the baseline TS2-Net, as many components (like the text parsing) can be pre-computed.

Limitations and Future Work

  • Parser Dependency: The method relies on the accuracy of off-the-shelf syntax parsers. If the parser fails to identify the correct grammatical structure (especially in informal or ungrammatical user queries), the hierarchical alignment may suffer.
  • Fixed Hyperparameters: The number of frames ($\lambda_{frame}$) and patches ($\lambda_{patch}$) to select are fixed hyperparameters. In reality, some actions take longer than others, and some entities are larger than others.
  • Future Direction: The authors suggest that integrating more complex reasoning, such as causal relationships between actions in the hierarchy, could further enhance retrieval for long-form videos.

Summary updated

Abstract

The user base of short video apps has experienced unprecedented growth in recent years, resulting in a significant demand for video content analysis. In particular, text-video retrieval, which aims to find the top matching videos given text descriptions from a vast video corpus, is an essential function, the primary challenge of which is to bridge the modality gap. Nevertheless, most existing approaches treat texts merely as discrete tokens and neglect their syntax structures. Moreover, the abundant spatial and temporal clues in videos are often underutilized due to the lack of interaction with text. To address these issues, we argue that using texts as guidance to focus on relevant temporal frames and spatial regions within videos is beneficial. In this paper, we propose a novel Syntax-Hierarchy-Enhanced text-video retrieval method (SHE-Net) that exploits the inherent semantic and syntax hierarchy of texts to bridge the modality gap from two perspectives. First, to facilitate a more fine-grained integration of visual content, we employ the text syntax hierarchy, which reveals the grammatical structure of text descriptions, to guide the visual representations. Second, to further enhance the multi-modal interaction and alignment, we also utilize the syntax hierarchy to guide the similarity calculation. We evaluated our method on four public text-video retrieval datasets of MSR-VTT, MSVD, DiDeMo, and ActivityNet. The experimental results and ablation studies confirm the advantages of our proposed method.
BibTeX
@article{yu2025she,
  author = {Xuzheng Yu and Chen Jiang and Xingning Dong and Tian Gan and Ming Yang and Qingpei Guo},
  title = {{SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrieval}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2025},
  doi = {10.1109/tcsvt.2025.3543840},
}

Loading related content...

Figure