Abstract
Traditional video recognition models often focus on a single dominant action, failing to capture the complexity of real-world scenarios where multiple actions occur simultaneously or sequentially. This paper proposes a Multi-Modal Multi-Action Graph Convolutional Network (GCN) framework that models the relationships between actions across visual, audio, and textual modalities. By treating actions as nodes in a graph and leveraging modality-specific GCNs, the model achieves state-of-the-art performance on the Multi-Moments in Time (M-MiT) benchmark, demonstrating significant improvements in both recognition accuracy and spatial localization.
Introduction
Most video recognition systems are optimized for single-label classification (e.g., "swimming"). However, real-world videos typically involve multiple concurrent or sequential actions, such as "performing," "dancing," and "singing." Recognizing these requires understanding the relational complexity between actions—for instance, "hitting a ball" often correlates with "running." Furthermore, while videos contain rich audio and textual metadata, these modalities are often underutilized, typically being fused only at the final stage of a model rather than informing the relational modeling of actions.
Methodology: Multi-Modal Multi-Action GCN
The proposed framework addresses these challenges by modeling action co-occurrences through a graph-based approach. Actions are represented as nodes, with edges reflecting their probability of co-occurrence.
Modality-Specific Branches
The architecture consists of three distinct branches:
- Visual GCN: Extracts spatiotemporal features using a 3D-CNN backbone (e.g., R(2+1)D).
- Audio GCN: Utilizes pre-computed audio embeddings (VGGish) from an "Audio Lexicon."
- Textual GCN: Employs word embeddings (GloVe or BERT) from a "Textual Lexicon."
Relational Modeling and Joint Learning
By processing these features through GCNs, the model can reason about action relationships. For example, if visual cues are ambiguous but audio clearly indicates "drumming," the Audio GCN reinforces the likelihood of related actions like "playing music." The entire system is trained end-to-end, ensuring that the visual backbone learns features that are inherently aware of multi-action relations.
Experiments and Results
The model was evaluated on the Multi-Moments in Time (M-MiT) dataset, which contains over 1 million videos with multiple labels per clip. Additional tests were conducted on the Charades and MultiTHUMOS datasets to verify generalizability.
Quantitative Performance
The framework achieved a state-of-the-art mean Average Precision (mAP) on M-MiT V1, outperforming existing benchmarks by approximately 3%. Ablation studies confirmed the value of multi-modality: adding audio and textual GCNs to a 3D-CNN baseline improved mAP from 54.8% to 58.7%.
Qualitative Analysis and Localization
Using Grad-CAM visualizations, the researchers demonstrated that the multi-modal approach leads to better spatial localization. Unlike baseline models that might focus on a single subject, this model identifies regions associated with multiple simultaneous actions.
Discussion and Conclusion
The core innovation of this work lies in treating actions as context for one another. By leveraging audio and text as "semantic anchors," the model becomes more robust to visual noise and occlusions. While the current implementation uses stationary lexicons, future work could explore dynamic, time-varying representations or Transformer-based architectures to further refine action-to-action correlation learning. Despite the added complexity, the model only introduces approximately 1.4M additional parameters, making it a computationally efficient upgrade for multi-action recognition tasks.
Summary updated