MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers

Muhammad Bilal Shaikh, Douglas Chai, Syed Mohammed Shamsul Islam, Naveed Akhtar

Introduction

Human Action Recognition (HAR) has traditionally been approached as a primarily visual task. However, many human activities—such as playing a musical instrument, clapping, or breaking objects—are characterized as much by their acoustic signatures as by their visual motion. Multimodal Human Action Recognition (MHAR) seeks to leverage these complementary signals to improve classification accuracy and robustness. Despite the potential, effectively fusing 1D audio signals with 3D video data presents significant architectural challenges.

Traditional methods often rely on Convolutional Neural Networks (CNNs) or Recurrent Neural Networks (RNNs). While effective, CNNs can be computationally prohibitive for high-resolution video, and RNNs often struggle with long-range temporal dependencies and limited parallelization. MAiVAR-T (Multimodal Audio-image and Video Action Recognizer using Transformers) addresses these limitations by introducing a unified transformer-based framework that treats audio signals as 2D images, enabling a streamlined and powerful fusion process.

Methodology

The MAiVAR-T framework is built upon an "audio-as-image" philosophy, which standardizes disparate data types into a format suitable for Vision Transformers.

Audio-to-Image Transmutation

Rather than processing raw audio waveforms, the model converts audio data into six distinct 2D visual representations:

  • Waveplots
  • Spectral Centroids
  • Spectral Rolloff
  • Mel-frequency Cepstral Coefficients (MFCCs)
  • MFCC Feature Scaling
  • Chromagrams

This transmutation allows the model to treat audio features as spatial data, making them compatible with standard vision architectures and allowing the system to benefit from the research momentum in Vision Transformers.

Dual-Stream Feature Extraction

MAiVAR-T utilizes a dual-stream architecture to process the synchronized modalities:

  1. Audio Stream: A standard Vision Transformer (ViT) processes the 2D audio-images.
  2. Video Stream: A Video Vision Transformer (ViViT) is employed to capture the complex spatiotemporal dynamics inherent in video sequences.

Fusion and Training

The features extracted from both streams are integrated using an AV-Fusion Multi-Layer Perceptron (MLP). This fusion layer leverages the self-attention mechanisms of the preceding transformers to weigh the importance of different modal features. The training is conducted in two stages: the first stage focuses on extracting robust unimodal features, while the second stage optimizes the cross-modal fusion and final classification.

Experimental Setup

The model was evaluated using a refined version of the UCF101 benchmark. Because approximately half of the original UCF101 videos lack audio, the researchers filtered the dataset to include only the 6,837 videos containing audio tracks. This subset, covering 51 action categories (referred to as UCF51), was split into 4,893 training samples and 1,944 testing samples to ensure a rigorous evaluation of the multimodal approach.

Results and Discussion

MAiVAR-T achieved a state-of-the-art accuracy of 91.2% on the UCF51 dataset, representing a significant improvement over existing benchmarks.

Performance Comparison

The transformer-based approach consistently outperformed both traditional 3D CNNs and previous multimodal frameworks:

  • MAiVAR-T: 91.2%
  • MAiVAR-CH (2022): 87.91%
  • MAFnet (2022): 86.72%
  • C3D (3D CNN): 82.23%

The Transformer Advantage

A critical finding of the study was the superiority of Transformer backbones over CNN backbones for processing audio-images. In ablation studies, replacing a CNN (such as InceptionResNet-v4) with a ViT backbone resulted in a 15% accuracy increase when using Spectral Centroid representations. This suggests that the global receptive field and self-attention mechanisms of Transformers are better suited for capturing the patterns within converted audio-images than the local receptive fields of standard convolutions.

Furthermore, the model demonstrated increased robustness to visual noise. Because audio-images are unaffected by environmental factors like lighting or camera angles, the multimodal representation provides a more stable foundation for recognition in real-world scenarios. Transformers also offer better complexity profiles regarding path length and parallelization compared to recurrent models, making them better suited for the long sequences found in video data.

Conclusion and Future Work

MAiVAR-T demonstrates that treating audio as a visual modality allows for a powerful, unified transformer architecture in human action recognition. By achieving 91.2% accuracy on UCF51, the model sets a new standard for multimodal fusion.

Future research will focus on scaling the architecture to larger datasets like Kinetics 400/600/700 and exploring the integration of additional modalities, such as text (speech-to-text or captions). Additionally, the authors plan to investigate generative AI-based transformer architectures to further refine the multimodal fusion process.

Summary updated

BibTeX
@inproceedings{shaikh2023maivar,
  author = {Muhammad Bilal Shaikh and Douglas Chai and Syed Mohammed Shamsul Islam and Naveed Akhtar},
  title = {{MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers}},
  booktitle = {EUVIP},
  year = {2023},
  doi = {10.1109/euvip58404.2023.10323051},
}
Figure