Text-to-Clip Video Retrieval with Early Fusion and Re-Captioning

Huijuan Xu, Kun He, Leonid Sigal, Stan Sclaroff, Kate Saenko

Abstract

We address the problem of text-based activity retrieval in video. Given a sentence describing an activity, our task is to retrieve matching clips from an untrimmed video. To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work. First, we inject text features early on when generating clip proposals, to help eliminate unlikely clips and thus speed up processing and boost performance. Second, to learn a fine-grained similarity metric for retrieval, we use visual features to modulate the processing of query sentences at the word level in a recurrent neural network. A multi-task loss is also employed by adding query re-generation as an auxiliary task. Our approach significantly outperforms prior work on two challenging benchmarks: Charades-STA and ActivityNet Captions.

Introduction

The task of temporal localization, or "text-to-clip" retrieval, involves identifying the exact start and end timestamps of a specific action within a long, untrimmed video based on a natural language query. For example, given a query like "person holding the refrigerator door open," a system must distinguish this specific moment from the rest of the video.

Most existing methods rely on "late fusion," where the entire sentence and the entire video clip are encoded into separate vectors before calculating a similarity score. This approach is often too coarse to capture fine-grained temporal details, such as the difference between "opening" and "closing" a door. Furthermore, traditional systems typically generate thousands of candidate clips regardless of the query, which is computationally expensive and introduces significant noise into the retrieval process.

Multilevel Integration Methodology

To overcome the limitations of late fusion, we propose a multilevel integration model that tightens the coupling between vision and language across three stages:

1. Query-Guided Segment Proposal Network (QSPN)

Instead of generating generic, query-agnostic clip proposals, our model uses the query's embedding to "attend" to video features early in the pipeline. By re-weighting convolutional features based on their relevance to the text before proposals are generated, the model narrows the search space to segments that are semantically relevant, improving both efficiency and accuracy.

2. Early Fusion Retrieval Model

Rather than comparing two static vectors, we employ a two-layer LSTM architecture for fine-grained similarity. The first layer embeds the query words, while the second layer processes these words while being conditioned on the visual features of a candidate clip at every single time step. This allows the model to learn how specific words (like verbs) relate to specific visual changes over time.

The early fusion retrieval model uses an LSTM to process query words while conditioned on visual features at each step.

3. Multi-task Loss (Re-Captioning)

To further strengthen the shared representation, we introduce an auxiliary "clip-to-text" task. During training, the model is required not only to retrieve the correct clip but also to "re-generate" the query sentence from the retrieved visual features. This re-captioning task acts as a powerful regularizer, ensuring the model truly understands the semantic relationship between the two modalities.

Experimental Results

We evaluated our model on two major benchmarks: Charades-STA, which focuses on indoor home activities, and ActivityNet Captions, a large-scale dataset originally designed for dense video captioning.

Our "Full Model" (combining LSTM, QSPN, and Captioning) consistently outperformed standard baselines:

  • Charades-STA: Achieved 54.7% Recall@1 (at tIoU=0.3), a significant improvement over the Vector Embedding (VE) baseline of 43.9%.
  • ActivityNet Captions: Reached 45.3% Recall@1 (at tIoU=0.3), compared to 39.3% for the VE baseline.

Key Insights

The Importance of Verbs

Objects like a "refrigerator" or a "person" are often present throughout a video, but the specific action (e.g., holding open vs. closing) defines the target clip. Early fusion allows the model to "look" at the video specifically through the lens of the verb being processed at that moment. This enables the system to correctly disambiguate actions that involve the same objects but different temporal dynamics.

Qualitative example of the model disambiguating actions like opening and closing a refrigerator door.

Efficiency and Regularization

By using the query to guide proposals via the QSPN, the model becomes more efficient by ignoring irrelevant video segments early on. Additionally, the re-captioning loss forces the LSTM to retain more semantic information about the video, which directly translates to superior retrieval performance.

Conclusion and Future Work

While our model shows significant gains, it can still be biased by specific phrases. For instance, a query containing "in the end" might lead the model to incorrectly localize a clip to the end of a video, regardless of the actual action. Future research will explore deeper modulation of visual feature extraction using language, such as through dynamic filters, to achieve even more precise vision-language alignment.

Summary updated

Abstract

We address the problem of text-based activity retrieval in video. Given a sentence describing an activity, our task is to retrieve matching clips from an untrimmed video. To capture the inherent structures present in both text and video, we introduce a multilevel model that integrates vision and language features earlier and more tightly than prior work. First, we inject text features early on when generating clip proposals, to help eliminate unlikely clips and thus speed up processing and boost performance. Second, to learn a fine-grained similarity metric for retrieval, we use visual features to modulate the processing of query sentences at the word level in a recurrent neural network. A multi-task loss is also employed by adding query re-generation as an auxiliary task. Our approach significantly outperforms prior work on two challenging benchmarks: Charades-STA and ActivityNet Captions.
BibTeX
@inproceedings{xu2019text,
  author = {Huijuan Xu and Kun He and Leonid Sigal and Stan Sclaroff and Kate Saenko},
  title = {{Text-to-Clip Video Retrieval with Early Fusion and Re-Captioning}},
  booktitle = {AAAI},
  year = {2019},
  url = {https://openreview.net/forum?id=SQKbaeMg_pH},
}

Loading related content...

Figure