Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification

Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, Syed Sameed Husain

Introduction

Video classification is a fundamentally multimodal task. To accurately categorize a video of a "concert," a model must process both the visual spectacle of the stage and the acoustic signature of the music. However, integrating these disparate data streams—audio and visual—is computationally demanding. State-of-the-art (SOTA) models typically rely on massive architectures with hundreds of millions of parameters to capture cross-modal nuances. While effective, these models are often too large for real-time applications or deployment on resource-constrained edge devices like smartphones.

This paper introduces Attend-Fusion, a compact architecture designed to bridge the gap between efficiency and performance. By leveraging hierarchical self-attention, Attend-Fusion achieves a significant reduction in model size while maintaining competitive accuracy on large-scale benchmarks.

Methodology: Hierarchical Attention and Fusion

The core innovation of Attend-Fusion lies in its hierarchical approach to feature refinement before the fusion step. Rather than simply concatenating raw audio and visual features, the model processes each modality through its own dedicated attention network.

Modality-Specific Self-Attention

Each modality (audio and visual) is first passed through fully connected (FC) layers, followed by a self-attention mechanism. This allows the model to "attend" to the most salient features within each stream independently. For instance, the audio network can learn to ignore background static, while the visual network focuses on primary objects rather than irrelevant background pixels. This acts as a powerful noise filter, ensuring that only high-quality representations are passed to the fusion stage.

Late Fusion and Refinement

After the individual modalities are refined, Attend-Fusion employs a "late fusion" strategy. The refined features are concatenated along the feature dimension. This joint representation then passes through additional refinement layers (FC layers) to capture complex cross-modal interactions. This combination of modality-specific attention followed by late fusion provides a robust balance between computational efficiency and predictive power.

Experimental Results

The researchers evaluated Attend-Fusion on the YouTube-8M dataset, a massive benchmark containing millions of videos with over 4,700 diverse visual entities. The models utilized pre-extracted video-level features from Inception networks (1024-dimensional for video, 128-dimensional for audio).

Efficiency vs. Accuracy

The quantitative results demonstrate a significant leap in efficiency:

  • Model Size: Attend-Fusion utilizes only 72 million parameters, representing an ~80% reduction in size compared to the heavy-weight baseline (FC Late Fusion at 341M parameters).
  • Predictive Performance: Despite the smaller footprint, Attend-Fusion achieved an F1 score of 75.64%, remarkably close to the 75.96% achieved by the much larger baseline.
  • Global Average Precision (GAP): A variant of the architecture, the Self-Attended Cross-Modal FCRN, achieved the highest overall GAP of 80.68%, outperforming all standard baselines.

Qualitative Precision

Beyond metrics, Attend-Fusion demonstrated superior precision in label assignment. In one instance, while a standard baseline predicted the generic label "Water" for a video of a water well, Attend-Fusion correctly identified the more specific and accurate labels "Pump" and "Water well."

Conclusion and Future Directions

Attend-Fusion proves that model complexity does not have a linear relationship with accuracy. By intelligently filtering information through self-attention, it is possible to build lightweight models that rival their massive counterparts.

While the results are promising, several avenues for future research remain. The current study relies on pre-extracted features; future iterations could explore end-to-end training to optimize the backbone encoders alongside the fusion mechanism. Additionally, while the model uses mean and standard deviation to represent temporal data, incorporating more explicit temporal modeling could further enhance performance. Finally, benchmarking the model's actual latency and power consumption on edge hardware will be crucial for validating its real-world utility in resource-constrained environments.

Summary updated

BibTeX
@inproceedings{awan2024attend,
  author = {Mahrukh Awan and Asmar Nadeem and Muhammad Junaid Awan and Armin Mustafa and Syed Sameed Husain},
  title = {{Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification}},
  booktitle = {Workshops @ ECCV},
  year = {2024},
  url = {https://openreview.net/forum?id=IvFN8kXlNJ},
}
Figure