Entity-aware and Motion-aware Transformers for Language-driven Action Localization

Shuo Yang, Xinxiao WuIJCAI 2022 · OpenReview

Abstract

Language-driven action localization in videos is a challenging task that involves not only visual-linguistic matching but also action boundary prediction. Recent progress has been achieved through aligning language queries to video segments, but estimating precise boundaries is still under-explored. In this paper, we propose entity-aware and motion-aware Transformers that progressively localize actions in videos by first coarsely locating clips with entity queries and then finely predicting exact boundaries in a shrunken temporal region with motion queries. The entity-aware Transformer incorporates the textual entities into visual representation learning via cross-modal and cross-frame attentions to facilitate attending action-related video clips. The motion-aware Transformer captures fine-grained motion changes at multiple temporal scales via integrating long short-term memory into the self-attention module to further improve the precision of action boundary prediction. Extensive experiments on the Charades-STA and TACoS datasets demonstrate that our method achieves better performance than existing methods.

Introduction: The Challenge of Boundary Precision

Language-driven action localization, also known as temporal video grounding, requires an AI to identify the exact start and end times of an activity described in natural language within an untrimmed video. While current models are proficient at general visual-linguistic matching, they often struggle with pinpointing precise boundaries.

Conceptual overview of entity-aware and motion-aware localization

The core issue is that standard models often treat the language query as a single semantic unit, failing to distinguish between static objects and the dynamic changes that define an action's boundaries. Human search strategies, by contrast, often decouple the "who/what" (entities) from the "how" (motion). Entities help identify the general location (coarse localization), while motion cues are critical for determining the exact temporal boundaries (fine localization).

The EAMAT Framework

The proposed Entity-aware and Motion-aware Transformer (EAMAT) architecture adopts a coarse-to-fine strategy to mimic this human-like search.

The EAMAT architecture overview

1. Entity-aware Transformer (Coarse Localization)

This module focuses on nouns and adjectives extracted from the query (e.g., "person", "mirror"). By using cross-modal and cross-frame attention, it scans the entire video to highlight clips where relevant entities appear. This step effectively narrows the search space to action-related segments.

2. Motion-aware Transformer (Fine Localization)

Once the candidate region is identified, the motion-aware module focuses on verbs and adverbs (e.g., "watching").

  • LSTM-Transformer Innovation: Standard Transformers are permutation-invariant and can struggle with local sequential motion. EAMAT addresses this by replacing standard linear projections in the self-attention mechanism with LSTM-based projections.
  • Multi-scale Temporal Modeling: By integrating Long Short-Term Memory (LSTM) cells into the self-attention module at multiple temporal scales, the model captures fine-grained motion changes that signal action boundaries more accurately than global attention alone.

Experimental Results

The model was evaluated on two major benchmarks: Charades-STA (daily indoor activities) and TACoS (cooking videos). EAMAT outperformed previous state-of-the-art (SOTA) methods on both datasets.

Dataset Metric EAMAT Performance Improvement over SOTA
Charades-STA R@1 (IoU=0.3) 74.19% +0.35% over SPA
TACoS R@1 (IoU=0.3) 50.11% +1.98% over CPN

Ablation studies confirmed the importance of both components. Removing the Entity-aware component dropped performance by about 3%, while replacing the LSTM-based motion module with standard layers caused a significant performance hit, proving that specialized motion modeling is the "secret sauce" for boundary precision.

Conclusion

EAMAT demonstrates that decoupling entity and motion information leads to superior action localization. By embedding LSTMs within the Transformer architecture, the model gains the ability to perceive fine-grained temporal flow while maintaining global context. While the model can be sensitive to Part-of-Speech (PoS) tagging errors, it sets a new standard for temporal grounding. Future work will explore extending this framework to weakly-supervised settings where explicit boundary annotations are unavailable during training.

Summary updated

Abstract

Language-driven action localization in videos is a challenging task that involves not only visual-linguistic matching but also action boundary prediction. Recent progress has been achieved through aligning language queries to video segments, but estimating precise boundaries is still under-explored. In this paper, we propose entity-aware and motion-aware Transformers that progressively localize actions in videos by first coarsely locating clips with entity queries and then finely predicting exact boundaries in a shrunken temporal region with motion queries. The entity-aware Transformer incorporates the textual entities into visual representation learning via cross-modal and cross-frame attentions to facilitate attending action-related video clips. The motion-aware Transformer captures fine-grained motion changes at multiple temporal scales via integrating long short-term memory into the self-attention module to further improve the precision of action boundary prediction. Extensive experiments on the Charades-STA and TACoS datasets demonstrate that our method achieves better performance than existing methods.
BibTeX
@inproceedings{yang2022entity,
  author = {Shuo Yang and Xinxiao Wu},
  title = {{Entity-aware and Motion-aware Transformers for Language-driven Action Localization}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=MkNgMN73pd},
}
Figure