Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification

Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, Syed Sameed Husain

TL;DR

Attend-Fusion is a compact audio-visual fusion model for video classification that slashes parameter counts by nearly 80% compared to heavy baseline models while maintaining competitive performance. By strategically combining self-attention mechanisms with a late-fusion pipeline, it achieves a 75.64% F1 score on the massive YouTube-8M dataset with only 72 million parameters, making high-performance multimodal AI viable for edge devices.

What Problem Does This Paper Address?

Video classification is inherently multimodal—both what we see and what we hear provide vital context for understanding a scene. However, effectively fusing audio and visual streams usually requires massive neural network architectures. This leads to high computational complexity and large memory footprints, making deployment on resource-constrained edge devices difficult or impossible.

Conversely, simply shrinking these architectures often tanks their performance because they fail to capture the intricate, fine-grained relationships between modalities. The authors address this trade-off by asking: Can we build a compact, parameter-efficient architecture that effectively leverages complementary multimodal information without sacrificing classification accuracy?

Methods

The authors introduce Attend-Fusion, alongside several other attention-based architectures, to optimize the trade-off between model size and accuracy.

Instead of relying purely on dense, oversized fully connected layers, Attend-Fusion utilizes a hierarchical approach:

  1. Modality-Specific Self-Attention: Audio and visual features are first processed independently through their own attention networks (fully connected layers coupled with self-attention mechanisms). This allows the model to dynamically prioritize the most salient features within each modality before they are mixed.
  2. Late Fusion via Concatenation: The attended audio and visual representations are then concatenated along the feature dimension.
  3. Cross-Modal Processing: The fused features pass through final fully connected layers to capture complex cross-modal interactions before predicting the multi-label outputs.
Fig. 1: Overview of our proposed audio-visual video classification framework on YouTube-8M dataset, illustrating different fusion mechanisms of audio and visual modalities [7,21,37,45].

To keep the architecture compact, the proposed models restrict the dimensions of their fully connected layers to 2K, compared to the massive 8K dimensions used in the baseline architectures.

Datasets

The models were trained and evaluated on the YouTube-8M dataset, a large-scale video benchmark containing millions of videos annotated with a vocabulary of 4,716 visual entities.

To ensure a fair comparison with prior work, the authors used pre-extracted, video-level features (Mean and Standard Deviation, or MF+STD) from pre-trained Inception networks:

  • Visual features: 1024-dimensional vectors.
  • Audio features: 128-dimensional vectors.

Key Insights and Ideas

  • Attention is a Parameter Saver: Incorporating self-attention allows the network to focus on what matters most, meaning the subsequent fully connected layers don't need to be as wide. This design choice is the primary driver behind the model's massive parameter reduction.
  • Late Fusion Wins on Efficiency: While complex cross-modal attention networks (where audio directly guides visual attention and vice versa) achieve slightly higher Global Average Precision (GAP), the late-fusion approach of Attend-Fusion yields the best balance, delivering top-tier F1 scores with a fraction of the parameters.
  • Multimodal Synergy: Unimodal baselines perform significantly worse (e.g., the Audio-only baseline achieves a dismal 50.32% GAP), proving that accurate video understanding relies heavily on the interplay between sight and sound.

Key Results

  • 80% Reduction in Model Size: Attend-Fusion achieves an F1 score of 75.64% with only 72M parameters. This is comparable to the best baseline model (Fully-Connected Late Fusion), which scores 75.96% F1 but requires a whopping 341M parameters.
  • Top Precision: The Self-Attended Cross-Modal FCRN Network achieved the highest Global Average Precision (80.68% GAP) among the proposed models, outperforming early-fusion baselines while using significantly fewer parameters (172M vs 341M+).
  • Ablation Validation: Removing the attention mechanism from Attend-Fusion caused the GAP to drop to 79.06% and the F1 score to plummet to 73.31%, proving that attention is the critical component enabling its compact efficiency.

Limitations and Future Work

While the paper demonstrates excellent efficiency gains, it relies entirely on pre-extracted features (MF+STD) rather than exploring end-to-end training directly from raw video pixels and audio waveforms. Future work could investigate how these compact attention mechanisms scale when integrated directly into raw feature extractors. Additionally, while the authors highlight the potential for deployment in resource-constrained environments, the paper does not provide hardware-specific profiling (such as actual latency, power consumption, or RAM usage on mobile or edge hardware).

Summary updated

Abstract

Exploiting both audio and visual modalities for video classification is a challenging task, as the existing methods require large model architectures, leading to high computational complexity and resource requirements. Smaller architectures, on the other hand, struggle to achieve optimal performance. In this paper, we propose Attend-Fusion, an audio-visual (AV) fusion approach that introduces a compact model architecture specifically designed to capture intricate audio-visual relationships in video data. Through extensive experiments on the challenging YouTube-8M dataset, we demonstrate that Attend-Fusion achieves an F1 score of 75.64\% with only 72M parameters, which is comparable to the performance of larger baseline models such as Fully-Connected Late Fusion (75.96\% F1 score, 341M parameters). Attend-Fusion achieves similar performance to the larger baseline model while reducing the model size by nearly 80\%, highlighting its efficiency in terms of model complexity. Our work demonstrates that the Attend-Fusion model effectively combines audio and visual information for video classification, achieving competitive performance with significantly reduced model size. This approach opens new possibilities for deploying high-performance video understanding systems in resource-constrained environments across various applications.
BibTeX
@inproceedings{awan2024attend,
  author = {Mahrukh Awan and Asmar Nadeem and Muhammad Junaid Awan and Armin Mustafa and Syed Sameed Husain},
  title = {{Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification}},
  booktitle = {Workshops @ ECCV},
  year = {2024},
  url = {https://openreview.net/forum?id=IvFN8kXlNJ},
}
Figure