Learning to Combine the Modalities of Language and Video for Temporal Moment Localization

Jungkyoo Shin, Jinyoung Moon

Abstract

Temporal moment localization (TML) aims to retrieve the best video segment matching a moment specified by a natural language query. Existing methods often generate visual and semantic embeddings independently, failing to fully consider long-term temporal relationships. We introduce a novel recurrent unit, Cross-Modal Long Short-Term Memory (CM-LSTM), which mimics human cognitive processes by focusing on query-related video segments while accumulating contextual information. Furthermore, we propose the Two-stream Attentive Cross-modal Interaction network (TACI), which employs a two-stream attention mechanism to preserve necessary visual information and generates precise 2D proposal maps by integrating global contextual features and local boundary scores. TACI outperforms state-of-the-art methods on ActivityNet-Captions with an R@1 of 45.50% ([email protected]) and achieves a significant 32x speedup in inference time.

Introduction

Temporal Moment Localization (TML) is a challenging task that requires a model to identify the start and end times of a specific activity within an untrimmed video based on a natural language description. As shown in Figure 1, queries can describe complex sequential actions or specific spatiotemporal intersections.

Examples of temporal moment localization tasks involving spatiotemporally intersected moments and sequential actions.

Current research in TML faces three primary hurdles:

  1. Inefficient Fusion: Visual and textual features are often fused late in the pipeline, missing complex temporal interactions.
  2. Information Loss: Standard attention mechanisms may discard visual context that is not directly mentioned in the query but is essential for boundary detection.
  3. Boundary Imprecision: Models often struggle to balance global context with local boundary details.

Methodology

To address these gaps, we propose the TACI framework, built upon three core innovations.

Cross-Modal LSTM (CM-LSTM)

The CM-LSTM is designed to integrate modalities recurrently. Unlike standard LSTMs, it accepts both video clips and query features at every timestep. A "modal gate" regulates information flow, allowing the model to refine its understanding of the query using video context and vice versa as it "watches" the video. This mimics the human process of keeping a query in mind while scanning a scene.

Two-Stream Attention Mechanism

To prevent the loss of crucial visual information, TACI processes two parallel streams. One stream is query-attended, focusing on visual elements directly relevant to the text. The second stream is unattended, preserving raw video features. By fusing these streams, the model maintains a global sense of the scene, which is vital for accurately localizing the boundaries of an event.

Local-Global Evaluation and 2D Proposal Maps

TACI generates a 2D proposal map where each coordinate represents a potential start and end time. This map is populated by combining global features (derived from the CM-LSTM) with local boundary-aware scores. This dual approach ensures that the model considers both the "big picture" and fine-grained frame-level transitions.

Experimental Results

The TACI framework was evaluated on the ActivityNet-Captions and Charades-STA benchmarks.

Performance and Efficiency

On ActivityNet-Captions, TACI achieved an R@1 ([email protected]) of 45.50%, surpassing the previous state-of-the-art, 2D-TAN. Crucially, TACI is significantly more efficient. In a test across over 17,000 videos, TACI completed localization in 26.51 seconds, compared to 862.10 seconds for 2D-TAN—a 32x speedup. This efficiency stems from a shallower convolutional architecture that does not sacrifice accuracy.

Generalizability of CM-LSTM

To test the utility of the CM-LSTM unit, we replaced standard LSTMs in other existing models (e.g., CMIN, ABLR). In all cases, the models showed performance improvements, suggesting that CM-LSTM is a robust, "plug-and-play" component for cross-modal tasks.

Conclusion

The TACI framework advances temporal moment localization by integrating language and video features through the novel CM-LSTM and a two-stream attention mechanism. By balancing global context with local boundary signals, TACI achieves state-of-the-art accuracy with remarkable computational efficiency. Future work will focus on adaptive proposal generation to better handle videos of varying lengths and exploring open-set scenarios.

Summary updated

Abstract

Temporal moment localization aims to retrieve the best video segment matching a moment specified by a query. The existing methods generate the visual and semantic embeddings independently and fuse them without full consideration of the long-term temporal relationship between them. To address these shortcomings, we introduce a novel recurrent unit, cross-modal long short-term memory (CM-LSTM), by mimicking the human cognitive process of localizing temporal moments that focuses on the part of a video segment related to the part of a query, and accumulates the contextual information across the entire video recurrently. In addition, we devise a two-stream attention mechanism for both attended and unattended video features by the input query to prevent necessary visual information from being neglected. To obtain more precise boundaries, we propose a two-stream attentive cross-modal interaction network (TACI) that generates two 2D proposal maps obtained globally from the integrated contextual features, which are generated by using CM-LSTM, and locally from boundary score sequences and then combines them into a final 2D map in an end-to-end manner. On the TML benchmark dataset, ActivityNet-Captions, the TACI outperform state-of-the-art TML methods with R@1 of 45.50% and 27.23% for [email protected] and [email protected], respectively. In addition, we show that the revised state-of-the-arts methods by replacing the original LSTM with our CM-LSTM achieve performance gains.
BibTeX
@article{shin2022learning,
  author = {Jungkyoo Shin and Jinyoung Moon},
  title = {{Learning to Combine the Modalities of Language and Video for Temporal Moment Localization}},
  journal = {Comput. Vis. Image Underst.},
  year = {2022},
  doi = {10.1016/j.cviu.2022.103375},
}
Figure