Abstract
Temporal moment localization (TML) aims to retrieve the best video segment matching a moment specified by a natural language query. Existing methods often generate visual and semantic embeddings independently, failing to fully consider long-term temporal relationships. We introduce a novel recurrent unit, Cross-Modal Long Short-Term Memory (CM-LSTM), which mimics human cognitive processes by focusing on query-related video segments while accumulating contextual information. Furthermore, we propose the Two-stream Attentive Cross-modal Interaction network (TACI), which employs a two-stream attention mechanism to preserve necessary visual information and generates precise 2D proposal maps by integrating global contextual features and local boundary scores. TACI outperforms state-of-the-art methods on ActivityNet-Captions with an R@1 of 45.50% ([email protected]) and achieves a significant 32x speedup in inference time.
Introduction
Temporal Moment Localization (TML) is a challenging task that requires a model to identify the start and end times of a specific activity within an untrimmed video based on a natural language description. As shown in Figure 1, queries can describe complex sequential actions or specific spatiotemporal intersections.
Current research in TML faces three primary hurdles:
- Inefficient Fusion: Visual and textual features are often fused late in the pipeline, missing complex temporal interactions.
- Information Loss: Standard attention mechanisms may discard visual context that is not directly mentioned in the query but is essential for boundary detection.
- Boundary Imprecision: Models often struggle to balance global context with local boundary details.
Methodology
To address these gaps, we propose the TACI framework, built upon three core innovations.
Cross-Modal LSTM (CM-LSTM)
The CM-LSTM is designed to integrate modalities recurrently. Unlike standard LSTMs, it accepts both video clips and query features at every timestep. A "modal gate" regulates information flow, allowing the model to refine its understanding of the query using video context and vice versa as it "watches" the video. This mimics the human process of keeping a query in mind while scanning a scene.
Two-Stream Attention Mechanism
To prevent the loss of crucial visual information, TACI processes two parallel streams. One stream is query-attended, focusing on visual elements directly relevant to the text. The second stream is unattended, preserving raw video features. By fusing these streams, the model maintains a global sense of the scene, which is vital for accurately localizing the boundaries of an event.
Local-Global Evaluation and 2D Proposal Maps
TACI generates a 2D proposal map where each coordinate represents a potential start and end time. This map is populated by combining global features (derived from the CM-LSTM) with local boundary-aware scores. This dual approach ensures that the model considers both the "big picture" and fine-grained frame-level transitions.
Experimental Results
The TACI framework was evaluated on the ActivityNet-Captions and Charades-STA benchmarks.
Performance and Efficiency
On ActivityNet-Captions, TACI achieved an R@1 ([email protected]) of 45.50%, surpassing the previous state-of-the-art, 2D-TAN. Crucially, TACI is significantly more efficient. In a test across over 17,000 videos, TACI completed localization in 26.51 seconds, compared to 862.10 seconds for 2D-TAN—a 32x speedup. This efficiency stems from a shallower convolutional architecture that does not sacrifice accuracy.
Generalizability of CM-LSTM
To test the utility of the CM-LSTM unit, we replaced standard LSTMs in other existing models (e.g., CMIN, ABLR). In all cases, the models showed performance improvements, suggesting that CM-LSTM is a robust, "plug-and-play" component for cross-modal tasks.
Conclusion
The TACI framework advances temporal moment localization by integrating language and video features through the novel CM-LSTM and a two-stream attention mechanism. By balancing global context with local boundary signals, TACI achieves state-of-the-art accuracy with remarkable computational efficiency. Future work will focus on adaptive proposal generation to better handle videos of varying lengths and exploring open-set scenarios.
Summary updated