Exploiting Visual Semantic Reasoning for Video-Text Retrieval

Zerun Feng, Zhimin Zeng, Caili Guo, Zheng LiIJCAI 2020 · DOI

TL;DR

This paper introduces ViSERN, a novel network that leverages Graph Convolutional Networks (GCNs) with a random walk mechanism to reason about semantic relationships between regions within video frames. This approach significantly improves video-text retrieval by capturing richer visual semantics, leading to state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Video-text retrieval, the task of finding relevant videos for a given text query (or vice-versa), is a challenging problem due to the inherent discrepancy between visual and linguistic representations. Existing methods often rely on extracting frame-level features and aggregating them, but they largely overlook the intricate semantic relationships between objects and regions within a frame. Humans, however, understand videos by observing these interactions and evolving semantic connections. This paper aims to bridge this gap by explicitly modeling and reasoning about these visual semantic relationships to achieve more accurate video-text alignment.

Methods

The core of the proposed method, ViSERN (Visual Semantic Enhanced Reasoning Network), lies in its novel approach to visual semantic reasoning:

  • Region Feature Extraction: First, frames are processed to detect salient regions using a bottom-up attention mechanism (based on Faster R-CNN). Features for these regions are extracted.
  • Semantic Correlation Graph Construction: For each frame, the detected regions are treated as vertices in a fully-connected graph. The edges represent the semantic relevance between regions, computed using a dot-product attention mechanism. This creates a semantic correlation graph for each frame.
  • Random Walk Rule-Based GCN: The key innovation is the application of Graph Convolutional Networks (GCNs) on these semantic graphs, guided by a random walk rule. This allows the model to reason about how regions interact and influence each other. The random walk mechanism prioritizes connections between regions with higher semantic relevance, effectively enhancing the representation of discriminative regions while suppressing redundancy. This is achieved by incorporating a random walk-based graph Laplacian into the GCN's spectral convolution operation.
  • Frame and Video Representation: The reasoning-enhanced region features are then aggregated (e.g., via mean pooling) to form frame-level features. These are further processed to generate video-level representations.
  • Common Space Learning: Finally, both video and text features are mapped into a common embedding space using a multi-level encoder, and a hinge-based triplet ranking loss is employed to learn the similarity between video-text pairs, focusing on hard negatives.

Datasets

The paper evaluates ViSERN on two widely-used public benchmark datasets for video-text retrieval:

  • MSR-VTT: A large dataset with 10,000 video clips, each annotated with 20 sentence captions.
  • MSVD: A dataset containing 1,970 video clips with approximately 40 multilingual captions per clip. The authors used English captions and selected five per clip.

The paper follows standard data splits for training, testing, and validation.

Key Insights and Ideas

  • Semantic Reasoning is Crucial: The paper strongly argues that understanding the semantic relationships between visual elements within a frame is fundamental for accurate video-text retrieval, a aspect often overlooked by previous methods.
  • Graphs as a Natural Representation: Representing frame regions as nodes in a semantic graph provides a powerful framework for modeling their interdependencies.
  • Random Walks Enhance GCNs: Incorporating a random walk mechanism into GCNs allows for a more principled way to propagate information across the graph, emphasizing important semantic connections and filtering out noise. This is a novel application of random walks in this context.
  • Focusing on Discriminative Regions: The random walk-based reasoning helps to highlight the most semantically relevant regions, leading to more discriminative frame representations.
  • Visualization Confirms Reasoning: The attention maps generated by the model visually demonstrate that ViSERN effectively captures pivotal semantic interactions and suppresses redundant information, aligning with human intuition.

Key Results

ViSERN achieves state-of-the-art performance on both the MSR-VTT and MSVD datasets for video-text retrieval.

  • MSR-VTT: ViSERN improves upon previous state-of-the-art methods by 2.6%, 4.5%, and 2.5% in R@1, R@5, and R@10 for text-to-video retrieval, respectively.
  • MSVD: The model also shows significant gains on this dataset.
  • Ablation Studies: Experiments confirm the effectiveness of the random walk rule and the GCN-based reasoning. Removing the reasoning component (ViSERNno-rea) significantly degrades performance, highlighting the importance of the proposed semantic reasoning.

Limitations and Future Work

  • Limitations: The paper acknowledges that the current graph construction is undirected. While effective, it doesn't explicitly model temporal dynamics between frames. The current method also focuses on spatial relationships within frames.
  • Future Work: The authors suggest several promising directions:
  • Incorporating location information of regions into the graph.
  • Extending the undirected graph to a directed graph to model video temporal sequences more effectively.
  • Further developing the model for content-based searches on online video websites.

Summary updated

BibTeX
@inproceedings{feng2020exploiting,
  author = {Zerun Feng and Zhimin Zeng and Caili Guo and Zheng Li},
  title = {{Exploiting Visual Semantic Reasoning for Video-Text Retrieval}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/140},
}
Figure