Asymmetric Cross-Guided Attention Network for Actor and Action Video Segmentation From Natural Language Query

Hao Wang, Cheng Deng, Junchi Yan, Dacheng Tao

Introduction

Actor and action video segmentation from natural language queries is a complex task that requires a model to identify and segment a specific actor performing a specific action based on a textual description. Traditional methods often rely on "dynamic convolution," where a text query is used to generate a convolutional filter. However, these methods are highly sensitive to linguistic variations—where different phrasings of the same intent produce different filters—and often lack the global context necessary to distinguish actors in complex scenes.

The ACAN Framework

To address these issues, the authors propose the Asymmetric Cross-Guided Attention Network (ACAN). Unlike previous approaches that use simple concatenation or dynamic filters, ACAN employs a dual-attention mechanism to allow for deep interaction between visual and linguistic features.

The overall architecture of the Asymmetric Cross-Guided Attention Network (ACAN), showing the multi-modal encoder, the attention network, and the multi-resolution decoder.

The pipeline begins with a multi-modal feature encoder. Video features are extracted using an Inflated 3D ConvNet (I3D) to capture spatio-temporal information. Simultaneously, the language query is processed using Word2Vec embeddings and a 1D CNN to capture semantic meaning. These features are then fed into the core of the model: the Asymmetric Cross-guided Attention Network.

Asymmetric Cross-Guided Attention

The central innovation of ACAN is its asymmetric approach to cross-modal attention. Rather than a symmetric co-attention, the model implements two distinct paths: Vision-Guided Language Attention (VGLA) and Language-Guided Vision Attention (LGVA).

Detailed view of the Asymmetric Cross-Guided Attention module, illustrating how video and language features are used to mutually refine each other through query, key, and value operations.

VGLA uses visual features to attend to the text, helping the model focus on the most relevant words and mitigating the impact of linguistic noise. LGVA then uses these refined linguistic features to highlight relevant pixels across the entire video frame, incorporating global context. This asymmetry ensures that vision helps "clean up" the language while language helps "locate" the actor within the vision, making the model significantly more robust to how a sentence is structured.

Experimental Results

The model was evaluated on the A2D Sentences and J-HMDB Sentences datasets. On the A2D benchmark, ACAN achieved a state-of-the-art Overall IoU of 60.1%, a significant improvement over the previous best of 55.1%. Notably, the model showed a five-fold increase in high-precision performance ([email protected]), reaching 2.0% compared to the previous 0.4%. Furthermore, the model demonstrated strong generalization capabilities, outperforming existing methods on the J-HMDB dataset without fine-tuning (57.6% vs 54.1% Overall IoU).

Conclusion

The Asymmetric Cross-Guided Attention Network represents a significant step forward in video segmentation from natural language. By moving beyond local convolutional filters to a global, asymmetric attention mechanism, the model achieves higher precision and greater robustness to linguistic variation. Future work will focus on improving zero-shot generalization and optimizing the computational efficiency of these complex attention mechanisms for real-time applications.

Summary updated

Abstract

Actor and action video segmentation from natural language query aims to selectively segment the actor and its action in a video based on an input textual description. Previous works mostly focus on learning simple correlation between two heterogeneous features of vision and language via dynamic convolution or fully convolutional classification. However, they ignore the linguistic variation of natural language query and have difficulty in modeling global visual context, which leads to unsatisfactory segmentation performance. To address these issues, we propose an asymmetric cross-guided attention network for actor and action video segmentation from natural language query. Specifically, we frame an asymmetric cross-guided attention network, which consists of vision guided language attention to reduce the linguistic variation of input query and language guided vision attention to incorporate query-focused global visual context simultaneously. Moreover, we adopt multi-resolution fusion scheme and weighted loss for foreground and background pixels to obtain further performance improvement. Extensive experiments on Actor-Action Dataset Sentences and J-HMDB Sentences show that our proposed approach notably outperforms state-of-the-art methods.
BibTeX
@inproceedings{wang2019asymmetric,
  author = {Hao Wang and Cheng Deng and Junchi Yan and Dacheng Tao},
  title = {{Asymmetric Cross-Guided Attention Network for Actor and Action Video Segmentation From Natural Language Query}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00404},
}
Figure