End-to-end audiovisual feature fusion for active speaker detection

Fiseha B. Tesema, Zheyuan Lin, Shiqiang Zhu, Wei Song, Jason Gu, Hong Wu

TL;DR

Researchers have developed a novel, two-stream end-to-end framework for Active Speaker Detection (ASD) that achieves state-of-the-art accuracy while remaining fast enough for real-time use. By fusing visual features from a VGG-M network with raw Mel Frequency Cepstrum Coefficients (MFCC) audio features, the model achieves 88.9% accuracy with an inference time of just 44.41 ms.

What Problem Does This Paper Address?

Active Speaker Detection (ASD) is the task of identifying which person (or persons) is speaking in a video at any given moment. This is a foundational step for technologies like human-robot interaction, automated video editing, and speech enhancement.

While deep learning has pushed the boundaries of ASD, existing models face two major hurdles:

  1. Complexity vs. Speed: Many high-performing models are too computationally "heavy" or require large input sizes, making them unsuitable for real-time applications.
  2. Feature Homogeneity: Most frameworks use Convolutional Neural Networks (ConvNets) to extract features from both video and audio. The authors argue that this standard approach hasn't fully explored the potential of fusing raw audio features directly with visual data.

Methods

The authors propose a two-stream end-to-end framework designed for efficiency:

  • Visual Stream: Uses a modified VGG-M architecture. Unlike standard versions, this one uses $5 \times 7 \times 7$ filters to better capture motion information across a sequence of five face frames.
  • Audio Stream: Instead of a complex ConvNet, the model uses MFCCs (Mel Frequency Cepstrum Coefficients)—a representation of sound based on human hearing. They tested three variations: raw MFCCs, MFCCs passed through a VGG-M, and MFCCs passed through fully connected (Fc) layers.
  • Temporal Modeling: Each stream is attached to a two-layer Bidirectional Gated Recurrent Unit (BiGRU) to handle the timing and dynamics of speech and movement.
  • Fusion: The outputs of both streams are concatenated and fed into a final BiGRU layer to model the joint relationship between sight and sound before a softmax layer provides the final "speaking/not speaking" label.

Datasets

The model was evaluated on the AVA-ActiveSpeaker dataset. This is a massive, challenging benchmark containing:

  • 3.65 million labeled face frames (approx. 38.5 hours of video).
  • Significant real-world "noise," such as low-resolution faces, occlusions (faces partially blocked), and profile views.
  • A diverse range of speakers across 113 training and 32 testing videos.

Key Insights and Ideas

  • Raw is Robust: One of the most surprising findings was that raw MFCC features are more robust to noise than features processed through additional ConvNet layers. When Gaussian noise was added to the audio, the model using raw MFCCs (M1) saw a smaller drop in accuracy compared to the more complex versions.
  • Simplicity Wins for Speed: By avoiding heavy ConvNets for the audio stream, the researchers significantly reduced the computational burden. This allows the model to process frames in under 45ms, crossing the threshold for real-time viability.
  • Temporal Context Matters: Using BiGRUs on each stream before fusion allows the model to understand the "rhythm" of each modality independently before trying to sync them up.

Key Results

  • Accuracy: The best-performing model (M3, which used a VGG-M on top of MFCCs) reached 88.929% accuracy, which is nearly identical to current state-of-the-art models that are often much slower.
  • Inference Speed: The "M1" model (using raw MFCCs) achieved a lightning-fast inference time of 44.41 ms. Even the most accurate version (M3) stayed under 48 ms.
  • Ablation Success: The study showed that moving from a single-layer BiGRU to a two-layer BiGRU improved detection accuracy by nearly 1%, proving the importance of deep temporal modeling in ASD.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Audio Feature Sophistication: While MFCCs are efficient and robust, the authors suggest that exploring even more sophisticated audio features could bridge the tiny remaining gap to the absolute state-of-the-art.
  • Hardware Specifics: The reported inference times were measured on a high-end NVIDIA RTX 2080Ti; performance on edge devices or mobile hardware remains an open question for future testing.

Summary updated

Abstract

Active speaker detection plays a vital role in human-machine interaction. Recently, a few end-to-end audiovisual frameworks emerged. However, these models' inference time was not explored and are not applicable for real-time applications due to their complexity and large input size. In addition, they explored a similar feature extraction strategy that employs the ConvNet on audio and visual inputs. This work presents a novel two-stream end-to-end framework fusing features extracted from images via VGG-M with raw Mel Frequency Cepstrum Coefficients features extracted from the audio waveform. The network has two BiGRU layers attached to each stream to handle each stream's temporal dynamic before fusion. After fusion, one BiGRU layer is attached to model the joint temporal dynamics. The experiment result on the AVA-ActiveSpeaker dataset indicates that our new feature extraction strategy shows more robustness to noisy signals and better inference time than models that employed ConvNet on both modalities. The proposed model predicts within 44.41 ms, which is fast enough for real-time applications. Our best-performing model attained 88.929% accuracy, nearly the same detection result as state-of-the-art work.
BibTeX
@inproceedings{tesema2022end,
  author = {Fiseha B. Tesema and Zheyuan Lin and Shiqiang Zhu and Wei Song and Jason Gu and Hong Wu},
  title = {{End-to-end audiovisual feature fusion for active speaker detection}},
  booktitle = {Fourteenth},
  year = {2022},
  doi = {10.1117/12.2643881},
}

Loading related content...

Figure