Introduction
Actor and action video segmentation from natural language queries is a complex task that requires a model to identify and segment a specific actor performing a specific action based on a textual description. Traditional methods often rely on "dynamic convolution," where a text query is used to generate a convolutional filter. However, these methods are highly sensitive to linguistic variations—where different phrasings of the same intent produce different filters—and often lack the global context necessary to distinguish actors in complex scenes.
The ACAN Framework
To address these issues, the authors propose the Asymmetric Cross-Guided Attention Network (ACAN). Unlike previous approaches that use simple concatenation or dynamic filters, ACAN employs a dual-attention mechanism to allow for deep interaction between visual and linguistic features.
The pipeline begins with a multi-modal feature encoder. Video features are extracted using an Inflated 3D ConvNet (I3D) to capture spatio-temporal information. Simultaneously, the language query is processed using Word2Vec embeddings and a 1D CNN to capture semantic meaning. These features are then fed into the core of the model: the Asymmetric Cross-guided Attention Network.
Asymmetric Cross-Guided Attention
The central innovation of ACAN is its asymmetric approach to cross-modal attention. Rather than a symmetric co-attention, the model implements two distinct paths: Vision-Guided Language Attention (VGLA) and Language-Guided Vision Attention (LGVA).
VGLA uses visual features to attend to the text, helping the model focus on the most relevant words and mitigating the impact of linguistic noise. LGVA then uses these refined linguistic features to highlight relevant pixels across the entire video frame, incorporating global context. This asymmetry ensures that vision helps "clean up" the language while language helps "locate" the actor within the vision, making the model significantly more robust to how a sentence is structured.
Experimental Results
The model was evaluated on the A2D Sentences and J-HMDB Sentences datasets. On the A2D benchmark, ACAN achieved a state-of-the-art Overall IoU of 60.1%, a significant improvement over the previous best of 55.1%. Notably, the model showed a five-fold increase in high-precision performance ([email protected]), reaching 2.0% compared to the previous 0.4%. Furthermore, the model demonstrated strong generalization capabilities, outperforming existing methods on the J-HMDB dataset without fine-tuning (57.6% vs 54.1% Overall IoU).
Conclusion
The Asymmetric Cross-Guided Attention Network represents a significant step forward in video segmentation from natural language. By moving beyond local convolutional filters to a global, asymmetric attention mechanism, the model achieves higher precision and greater robustness to linguistic variation. Future work will focus on improving zero-shot generalization and optimizing the computational efficiency of these complex attention mechanisms for real-time applications.
Summary updated