Abstract
Recent methods in text-to-video moment retrieval using natural language queries typically require full supervision during training. However, acquiring temporal boundary annotations for large-scale video datasets is time-consuming and often unscalable. We introduce a weakly supervised framework for this task, where training relies only on video-level sentence descriptions without explicit temporal extents. Our approach utilizes a joint visual-semantic embedding framework and Text-Guided Attention (TGA) to discover latent alignments between video frames and text. Experiments on benchmark datasets show that our weakly supervised method achieves performance comparable to state-of-the-art fully supervised approaches.
Introduction
Video Moment Retrieval involves identifying the specific start and end times of an action within a long video based on a natural language query. While effective, traditional fully supervised models require expensive human-annotated temporal boundaries. This work explores a weakly supervised alternative, learning to localize moments using only the knowledge that a specific description applies to some part of a given video.
Methodology: Text-Guided Attention
The proposed framework maps video and text features into a shared high-dimensional embedding space. The core innovation is Text-Guided Attention (TGA), which calculates the cosine similarity between the text query and individual video time steps.
- Feature Extraction: Visual features are extracted using pre-trained networks (C3D or VGG16), while text queries are encoded via a Gated Recurrent Unit (GRU).
- Latent Alignment: Similarity scores are processed through a softmax layer to generate an attention map, identifying relevant video segments.
- Training: The model is optimized using a hinge-based triplet ranking loss, ensuring that a video's correct description is closer to its visual features than mismatched descriptions.
During inference, the peaks in the TGA attention map serve as the predicted temporal boundaries for retrieval.
Experimental Evaluation
The model was tested on two benchmarks: Charades-STA (16,000+ pairs of indoor activities) and DiDeMo (27,000+ moments with a 5-second segment structure).
Key Results
On Charades-STA, the method achieved a Recall@1 (IoU 0.5) of 19.94%, outperforming visual-semantic baselines like VSA-STV (16.91%). On DiDeMo, the model reached an mIoU of 24.92, nearly matching the fully supervised LSTM-RGB-Local baseline (25.13). These results suggest that video-level labels provide sufficient signal for effective temporal localization.
Conclusion and Future Work
This research demonstrates that attention mechanisms can serve as both a weighting tool and a retrieval mechanism. By learning to satisfy a global ranking loss, the model implicitly learns to localize specific moments. Future improvements may involve incorporating audio cues or spatial-temporal modeling to better handle complex contexts and small objects.
Summary updated