Text-Based Localization of Moments in a Video Corpus

Sudipta Paul, Niluthpol Chowdhury Mithun, Amit K. Roy-Chowdhury

TL;DR

This paper introduces a new challenge: finding a specific moment described by a text query within a large collection of videos, not just within a single pre-selected video. They propose the Hierarchical Moment Alignment Network (HMAN) to tackle this by learning a joint embedding space that aligns video moments and sentences, considering both local activity details and global video semantics.

What Problem Does This Paper Address?

Traditional video moment localization systems assume you already know which video contains the moment you're looking for. They then focus on pinpointing the exact start and end times of that moment within that specific video. However, in real-world scenarios, users often have a text query (e.g., "a person making coffee") and need to find that moment across a vast library of videos. This requires a two-step process: first, identifying the correct video from the corpus, and second, localizing the moment within that video. Existing methods are not designed for this "video corpus" scenario, as they assume a one-to-one correspondence between a query and a video.

Methods

The core of the proposed solution is the Hierarchical Moment Alignment Network (HMAN). HMAN addresses the problem by:

  • Joint Embedding Space: Learning to represent both video moments and sentences in a shared embedding space. This allows for direct comparison and alignment.
  • Hierarchical Moment Encoder: Instead of processing clips independently, HMAN uses stacked temporal convolutional layers in a hierarchical structure. This efficiently generates representations for candidate moments across different temporal scales within a video, capturing both short-term actions and longer event dynamics.
  • Dual Learning Objective: HMAN employs a novel learning objective that simultaneously addresses two key aspects:
  • Intra-video Moment Alignment: Learning to distinguish subtle differences between moments within the same video that are relevant to the query versus those that are not. This is achieved using triplet ranking losses.
  • Inter-video Global Semantics: Learning to distinguish between different videos based on their overall semantic content. This is crucial for retrieving the correct video from a corpus and is achieved by learning a "video-sentence relevance" score.
  • Feature Extraction: Standard pre-trained models are used for extracting visual features from video clips (e.g., ResNet-152) and text features from sentences (GloVe embeddings followed by Bi-directional GRU).

Datasets

The proposed method is evaluated on three benchmark datasets:

  • DiDeMo: Distinct Describable Moments dataset.
  • Charades-STA: A dataset for temporal localization of moments.
  • ActivityNet Captions: A dataset originally designed for dense video captioning.

These datasets consist of unsegmented, untrimmed videos with natural language sentence annotations and temporal grounding information.

Key Insights and Ideas

  • The "Video Corpus" is the Real Challenge: The paper convincingly argues that localizing moments within a corpus is a significantly harder and more practical problem than localizing within a single video.
  • Global Semantics Matter for Video Retrieval: Simply focusing on local moment-sentence alignment isn't enough. Distinguishing between the overall themes or "global semantics" of different videos is essential for correctly identifying the target video.
  • Hierarchical Encoding is Efficient: The hierarchical temporal convolutional network allows for efficient generation of moment representations across multiple scales, avoiding computationally expensive sliding window approaches.
  • Joint Learning is Powerful: Combining intra-video moment discrimination with inter-video global semantic learning leads to substantial performance gains, demonstrating the synergy between these two aspects.

Key Results

  • HMAN achieves state-of-the-art or highly competitive performance across all three datasets (DiDeMo, Charades-STA, ActivityNet Captions) for the task of text-based moment localization in a video corpus.
  • On the DiDeMo dataset, HMAN shows an absolute improvement of up to 11.88% in R@k metrics compared to baseline approaches.
  • The hierarchical moment encoder module significantly improves performance, with gains of around 14% (sum-margin) and 8% (max-margin) on DiDeMo for R@100, IoU=0.50.
  • The proposed video-sentence relevance computation using LogSumExp pooling outperforms average pooling.
  • Qualitative results (t-SNE visualizations and example retrievals) demonstrate that HMAN effectively aligns text queries with correct moments and clusters similar moments from the same video.

Limitations and Future Work

  • Computational Cost: While HMAN is more efficient than sliding window methods, inference still involves computing cosine similarity between query sentences and all candidate moments in the corpus, which can be computationally intensive for very large corpora.
  • Parameter Sensitivity: The paper notes that the parameter $\lambda_1$, which balances intra-video and inter-video learning, shows stable performance within a certain range ({3, 4, 5, 6, 7}). Further exploration of optimal $\lambda_1$ values or adaptive methods could be beneficial.
  • Future Work: The authors suggest exploring more advanced feature extraction techniques, investigating different loss functions, and potentially extending the work to handle more complex queries or longer video sequences. The paper also hints at the potential for improved performance with decreasing numbers of test set moment-sentence pairs, suggesting that the model's ability to generalize is strong.

Summary updated

Abstract

Prior works on text-based video moment localization focus on temporally grounding the textual query in an untrimmed video. These works assume that the relevant video is already known and attempt to localize the moment on that relevant video only. Different from such works, we relax this assumption and address the task of localizing moments in a corpus of videos for a given sentence query. This task poses a unique challenge as the system is required to perform: 2) retrieval of the relevant video where only a segment of the video corresponds with the queried sentence, 2) temporal localization of moment in the relevant video based on sentence query. Towards overcoming this challenge, we propose Hierarchical Moment Alignment Network (HMAN) which learns an effective joint embedding space for moments and sentences. In addition to learning subtle differences between intra-video moments, HMAN focuses on distinguishing inter-video global semantic concepts based on sentence queries. Qualitative and quantitative results on three benchmark text-based video moment retrieval datasets - Charades-STA, DiDeMo, and ActivityNet Captions - demonstrate that our method achieves promising performance on the proposed task of temporal localization of moments in a corpus of videos.
BibTeX
@article{paul2021text,
  author = {Sudipta Paul and Niluthpol Chowdhury Mithun and Amit K. Roy-Chowdhury},
  title = {{Text-Based Localization of Moments in a Video Corpus}},
  journal = {IEEE Trans. Image Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=3Ge9ihv-Bh},
}
Figure