Retrieving and Highlighting Action with Spatiotemporal Reference

Seito Kasai, Yuchi Ishikawa, Masaki Hayashi, Yoshimitsu Aoki, Kensho Hara, Hirokatsu Kataoka

Introduction

Cross-modal video retrieval—the task of finding a video based on a natural language description—faces a significant "abstraction gap." While videos consist of low-level pixel data across time, captions provide high-level semantic summaries. Traditional models often treat this as a "black box" problem, projecting both modalities into a shared embedding space. This approach typically lacks interpretability and fails to capture fine-grained distinctions between objects (nouns) and actions (verbs).

This paper introduces a framework designed to bridge this gap by not only retrieving the correct video but also identifying exactly where and when the described action occurs. The authors propose a new task called Action Highlighting, which requires generating spatiotemporal "voxels" that link specific words in a caption to relevant regions in the video.

Methodology

The proposed framework treats videos and text as multi-faceted entities, employing a dual-stream approach to align linguistic components with visual and temporal features.

System architecture showing dual-stream video processing and linguistic decomposition.

Dual-Stream Video Processing

The model utilizes a SlowFast Network to process video input. The "Slow" branch operates at high spatial resolution to capture fine-grained details necessary for identifying objects (nouns). Conversely, the "Fast" branch captures high-frequency motion data, which is critical for recognizing actions (verbs).

Linguistic Decomposition and Alignment

Rather than encoding the entire caption as a single vector, the system uses Part-of-Speech (POS) tagging to extract nouns and verbs. These are processed by separate Token Encoders:

  • Visual Space: Nouns are mapped here to align with spatial features from the Slow branch.
  • Motion Space: Verbs are mapped here to align with temporal features from the Fast branch.

The core innovation lies in the Spatiotemporal Alignment module. The model calculates a relevance map ($M$) by comparing local video features to the noun and verb embeddings. A Weighted Triplet Loss is then applied to ensure that the model "highlights" the correct regions. Notably, this training is performed under weak supervision; the model does not require manual, frame-by-frame annotations but instead learns these associations directly from raw video-caption pairs.

Key Results and Discussion

The framework was evaluated on the MSR-VTT and Kinetics-700 datasets. On MSR-VTT, the inclusion of motion and visual alignment led to a 2-3% improvement in Recall@1 compared to baselines lacking this explicit decomposition.

Action Highlighting and Interpretability

A key advantage of this approach is its interpretability. Unlike traditional saliency methods like Grad-CAM, which are often deterministic, this model provides verb-conditioned highlighting.

Comparison of spatiotemporal highlighting between Grad-CAM and the proposed method.

As shown in the qualitative results, the model can shift its focus based on the specific query token. For a caption involving a "woman smiling," the model highlights the person when queried with "woman" but focuses specifically on facial motion when queried with "smile." This demonstrates that the model learns to associate linguistic concepts with their corresponding spatiotemporal signatures rather than just the general "gist" of the scene.

Conclusion

By treating captions as spatiotemporal references, this framework achieves superior retrieval performance while providing visible "reasoning" for its outputs. While the model occasionally relies on contextual correlations (e.g., highlighting a street when the action is "ride"), it represents a significant step toward more transparent and fine-grained video understanding. Future work will explore more complex linguistic structures to further distinguish between objects and the actions performed upon them.

Summary updated

Abstract

In this paper, we present a framework that jointly retrieves and spatiotemporally highlights actions in videos by enhancing current deep cross-modal retrieval methods. Our work takes on the novel task of action highlighting, which visualizes where and when actions occur in an untrimmed video setting. Action highlighting is a fine-grained task, compared to conventional action recognition tasks which focus on classification or window-based localization. Leveraging weak supervision from annotated captions, our framework acquires spatiotemporal relevance maps and generates local embeddings which relate to the nouns and verbs in captions. Through experiments, we show that our model generates various maps conditioned on different actions, in which conventional visual reasoning methods only go as far as to show a single deterministic saliency map. Also, our model improves retrieval recall over our baseline without alignment by 2-3% on the MSR-VTT dataset.
BibTeX
@inproceedings{kasai2020retrieving,
  author = {Seito Kasai and Yuchi Ishikawa and Masaki Hayashi and Yoshimitsu Aoki and Kensho Hara and Hirokatsu Kataoka},
  title = {{Retrieving and Highlighting Action with Spatiotemporal Reference}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=vVgI4LgsFx5},
}
Figure