Multilevel Language and Vision Integration for Text-to-Clip Retrieval.

Huijuan Xu, Kun He, Bryan A. Plummer, Leonid Sigal, Stan Sclaroff, Kate Saenko

Abstract

We address the problem of text-based activity retrieval in video. Given a sentence describing an activity, our task is to retrieve matching clips from an untrimmed video. To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work. First, we inject text features early on when generating clip proposals, to help eliminate unlikely clips and thus speed up processing and boost performance. Second, to learn a fine-grained similarity metric for retrieval, we use visual features to modulate the processing of query sentences at the word level in a recurrent neural network. A multi-task loss is also employed by adding query re-generation as an auxiliary task. Our approach significantly outperforms prior work on two challenging benchmarks: Charades-STA and ActivityNet Captions.

Introduction

The task of temporal activity retrieval involves identifying the exact start and end timestamps of an activity in a long, untrimmed video based on a natural language query (e.g., "person holding the refrigerator door open"). Most existing methods rely on "late fusion," where the entire sentence and video clip are projected into a shared space for distance measurement. However, this coarse approach often loses the fine-grained relationships between specific words and visual movements, making it difficult to distinguish between similar actions like "opening" versus "closing" a door.

Methodology

To overcome the limitations of late fusion, we propose a multilevel integration framework that fuses vision and language features at multiple stages of the retrieval pipeline.

Query-Guided Segment Proposal Network (QSPN)

Traditional methods often generate thousands of "blind" clip proposals before considering the query. In contrast, our QSPN uses the text query to immediately "attend" to relevant parts of the video. By re-weighting visual features based on their similarity to the query embedding, the model ensures that candidate clips are contextually relevant from the outset, improving both efficiency and accuracy.

Early Fusion Retrieval Model

Once candidate clips are identified, a two-layer LSTM calculates a similarity score. The first layer embeds the query sentence, while the second layer processes the sentence word-by-word, conditioned on the visual features of the clip. This allows the model to dynamically associate specific words with specific visual cues.

The multilevel model uses an LSTM to integrate visual features with word-level query processing for fine-grained retrieval.

Multi-Task Loss

We incorporate a "query re-generation" task as an auxiliary loss ($L_{CAP}$). During training, the model is tasked with re-writing the query based on the retrieved clip. This acts as a powerful regularizer, ensuring the learned features are descriptive and capture the essential semantic relationship between the two modalities.

Experiments and Results

We evaluated our model on two major benchmarks: Charades-STA (daily indoor activities) and ActivityNet Captions (large-scale diverse videos).

  • Charades-STA: At a temporal overlap threshold of tIoU=0.5, our model achieved a Recall@1 of 35.6%, significantly outperforming the previous state-of-the-art (CTRL) at 23.6%.
  • ActivityNet Captions: The model achieved a Recall@1 of 45.3% (at tIoU=0.3), demonstrating robustness across complex video data.

Qualitative results show that our early-fusion model correctly disambiguates actions based on temporal ordering, whereas late-fusion models often fail when the objects involved are the same.

Discussion and Conclusion

Early fusion is critical for capturing the nuances of temporal activities. By letting visual features modulate query processing at the word level and using text to filter video segments early on, we achieve superior retrieval performance. While word-level modulation increases computational cost compared to simple dot-product similarity, the gains in accuracy are substantial. Future work will explore language modulation at the convolutional feature extraction level to further deepen the integration.

Summary updated

Abstract

We address the problem of text-based activity retrieval in video. Given a sentence describing an activity, our task is to retrieve matching clips from an untrimmed video. To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work. First, we inject text features early on when generating clip proposals, to help eliminate unlikely clips and thus speed up processing and boost performance. Second, to learn a fine-grained similarity metric for retrieval, we use visual features to modulate the processing of query sentences at the word level in a recurrent neural network. A multi-task loss is also employed by adding query re-generation as an auxiliary task. Our approach significantly outperforms prior work on two challenging benchmarks: Charades-STA and ActivityNet Captions.
BibTeX
@inproceedings{xu2019multilevel,
  author = {Huijuan Xu and Kun He and Bryan A. Plummer and Leonid Sigal and Stan Sclaroff and Kate Saenko},
  title = {{Multilevel Language and Vision Integration for Text-to-Clip Retrieval.}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=SQKbaeMg_pH},
}
Figure