Multi-Modal Multi-Action Video Recognition

Zhensheng Shi, Ju Liang, Qianqian Li, Haiyong Zheng, Zhaorui Gu, Junyu Dong, Bing ZhengICCV 2021 · DOI

Abstract

Traditional video recognition models often focus on a single dominant action, failing to capture the complexity of real-world scenarios where multiple actions occur simultaneously or sequentially. This paper proposes a Multi-Modal Multi-Action Graph Convolutional Network (GCN) framework that models the relationships between actions across visual, audio, and textual modalities. By treating actions as nodes in a graph and leveraging modality-specific GCNs, the model achieves state-of-the-art performance on the Multi-Moments in Time (M-MiT) benchmark, demonstrating significant improvements in both recognition accuracy and spatial localization.

Introduction

Most video recognition systems are optimized for single-label classification (e.g., "swimming"). However, real-world videos typically involve multiple concurrent or sequential actions, such as "performing," "dancing," and "singing." Recognizing these requires understanding the relational complexity between actions—for instance, "hitting a ball" often correlates with "running." Furthermore, while videos contain rich audio and textual metadata, these modalities are often underutilized, typically being fused only at the final stage of a model rather than informing the relational modeling of actions.

Methodology: Multi-Modal Multi-Action GCN

The proposed framework addresses these challenges by modeling action co-occurrences through a graph-based approach. Actions are represented as nodes, with edges reflecting their probability of co-occurrence.

Modality-Specific Branches

The architecture consists of three distinct branches:

  1. Visual GCN: Extracts spatiotemporal features using a 3D-CNN backbone (e.g., R(2+1)D).
  2. Audio GCN: Utilizes pre-computed audio embeddings (VGGish) from an "Audio Lexicon."
  3. Textual GCN: Employs word embeddings (GloVe or BERT) from a "Textual Lexicon."
The Multi-Modal Multi-Action GCN architecture, featuring spatiotemporal, audio, and textual branches integrated through joint learning.

Relational Modeling and Joint Learning

By processing these features through GCNs, the model can reason about action relationships. For example, if visual cues are ambiguous but audio clearly indicates "drumming," the Audio GCN reinforces the likelihood of related actions like "playing music." The entire system is trained end-to-end, ensuring that the visual backbone learns features that are inherently aware of multi-action relations.

Experiments and Results

The model was evaluated on the Multi-Moments in Time (M-MiT) dataset, which contains over 1 million videos with multiple labels per clip. Additional tests were conducted on the Charades and MultiTHUMOS datasets to verify generalizability.

Quantitative Performance

The framework achieved a state-of-the-art mean Average Precision (mAP) on M-MiT V1, outperforming existing benchmarks by approximately 3%. Ablation studies confirmed the value of multi-modality: adding audio and textual GCNs to a 3D-CNN baseline improved mAP from 54.8% to 58.7%.

Qualitative Analysis and Localization

Using Grad-CAM visualizations, the researchers demonstrated that the multi-modal approach leads to better spatial localization. Unlike baseline models that might focus on a single subject, this model identifies regions associated with multiple simultaneous actions.

Grad-CAM visualizations comparing a standard baseline with the proposed multi-modal model, showing improved localization across multiple actions.

Discussion and Conclusion

The core innovation of this work lies in treating actions as context for one another. By leveraging audio and text as "semantic anchors," the model becomes more robust to visual noise and occlusions. While the current implementation uses stationary lexicons, future work could explore dynamic, time-varying representations or Transformer-based architectures to further refine action-to-action correlation learning. Despite the added complexity, the model only introduces approximately 1.4M additional parameters, making it a computationally efficient upgrade for multi-action recognition tasks.

Summary updated

BibTeX
@inproceedings{shi2021multi,
  author = {Zhensheng Shi and Ju Liang and Qianqian Li and Haiyong Zheng and Zhaorui Gu and Junyu Dong and Bing Zheng},
  title = {{Multi-Modal Multi-Action Video Recognition}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01342},
}
Figure