Event-Specific Audio-Visual Fusion Layers: A Simple and New Perspective on Video Understanding

Arda Senocak, Junsik Kim, Tae-Hyun Oh, Dingzeyu Li, In So Kweon

Introduction

In video understanding, the integration of audio and visual modalities—known as "fusion"—is a critical challenge. Conventional approaches often rely on a "one-size-fits-all" strategy, typically employing global average pooling to aggregate features across an entire video clip. This method implicitly assumes that audio and visual signals are consistently correlated and continuous throughout the duration of the clip.

Comparison of the proposed multi-head approach versus prior single-head methods across diverse event types.

However, real-world events are heterogeneous: some are fleeting (instant), some are rhythmic, and others are persistent (continuous background noise). By forcing diverse event types through a single fusion bottleneck, existing models often "smooth out" critical, short-lived signals. This paper introduces Event-Specific Fusion Layers, a multi-head architecture designed to capture these nuanced relationships through specialized "expert" layers.

Methodology

The proposed framework moves away from monolithic fusion by utilizing a multi-head structure where each head is a parameter-free Event-Specific Layer designed to capture a specific type of interaction.

Architecture Overview

The system utilizes a two-stream backbone: an MC3-18 network for video and a modified convolutional network for audio. These extract high-level feature representations, $z^V$ and $z^A$, respectively, which are then passed to the fusion module.

The multi-head architecture showing the flow from video and audio networks to specialized event-specific layers.

Specialized Fusion Layers

The core innovation lies in five distinct layers that process the extracted features:

  1. Continuous Event Layer: Mimics standard fusion by averaging features across the entire temporal dimension.
  2. Instant Event Layer: Identifies specific moments of high audio-visual correlation (dot product) and pools features only from the top-$k$ most relevant time steps.
  3. Onset Event Layer: Targets the sudden initiation of sounds, which is particularly effective for rhythmic or percussive events like clapping or music.
  4. Visual-only & Audio-only Layers: These provide a fallback mechanism, allowing the model to prioritize a single modality when the other is uninformative (e.g., a silent handshake).

Training and Inference

The network is trained using a multi-task joint learning objective where every head predicts the same ground-truth label. During inference, the model aggregates predictions via majority voting or by selecting the head with the highest confidence score.

Experimental Evaluation

The model was evaluated across several benchmarks, including VGGSound, Kinetics-Sound, Kinetics-400, AVE (Audio-Visual Event), and LLP (Look, Listen, and Parse). Additionally, the authors introduced a Multi-labeled VGGSound subset to test the model's ability to discover secondary labels within a single video.

Key Results

  • Performance Boost: The approach yielded a ~2% improvement in accuracy over "Naïve AV" baselines on audio-visual intensive datasets like VGGSound and AVE.
  • Multi-label Discovery: On the LLP dataset, the model achieved an F1 score of 0.72, significantly outperforming standard models (which ranged from 0.50 to 0.66). Remarkably, the different heads often discovered distinct, valid labels (e.g., identifying "singing" and "guitar" separately) even when trained on a single label.
  • Interpretability and Localization: The model provides inherent interpretability; the most confident head indicates the nature of the event. Furthermore, it enables zero-shot sound source localization as a natural byproduct of its training.

Conclusion

Event-Specific Fusion Layers offer a simple yet powerful alternative to global pooling by acknowledging the diversity of audio-visual relationships. While the current implementation relies on hand-crafted categories, it demonstrates significant improvements in both performance and interpretability, paving the way for more nuanced video understanding models.

Summary updated

Abstract

To understand our surrounding world, our brain is continuously inundated with multisensory information and their complex interactions coming from the outside world at any given moment. While processing this information might seem effortless for human brains, it is challenging to build a machine that can perform similar tasks since complex interactions cannot be dealt with a single type of integration but require more sophisticated approaches. In this paper, we propose a new simple method to address the multisensory integration in video understanding. Unlike previous works where a single fusion type is used, we design a multi-head model with individual event-specific layers to deal with different audio-visual relationships, enabling different ways of audio-visual fusion. Experimental results show that our event-specific layers can discover unique properties of the audio-visual relationships in the videos, e.g., semantically matched moments, and rhythmic events. Moreover, although our network is trained with single labels, our multi-head design can inherently output additional semantically meaningful multi-labels for a video. As an application, we demonstrate that our proposed method can expose the extent of event-characteristics of popular benchmark datasets.
BibTeX
@inproceedings{senocak2023event,
  author = {Arda Senocak and Junsik Kim and Tae-Hyun Oh and Dingzeyu Li and In So Kweon},
  title = {{Event-Specific Audio-Visual Fusion Layers: A Simple and New Perspective on Video Understanding}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  doi = {10.1109/wacv56688.2023.00227},
}
Figure