QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge

Hsi-Che Lin, Chien-Yi Wang, Min-Hung Chen, Szu-Wei Fu, Yu-Chiang Frank WangarXiv 2023 · DOI · arXiv

Abstract

The Ego4D "Talking to Me" (TTM) challenge requires identifying if a person in an egocentric video is addressing the camera wearer. This task is complicated by significant noise in the Ego4D dataset, where many frames lack visual bounding boxes despite active vocalization. We present QuAVF, a data-centric approach that disentangles audio and visual processing. By introducing a facial-landmark-based "quality score," QuAVF intelligently filters training data and dynamically weights multi-modal fusion during inference. QuAVF achieved first place in the 2023 TTM challenge with a 67.4% mAP.

Introduction

The "Talking to Me" (TTM) task is a foundational capability for social robotics and augmented reality, requiring a system to determine social engagement from a first-person perspective. However, the Ego4D dataset presents a unique challenge: ground-truth labels are based on vocal activity, but approximately 40% of labeled frames (0.7M out of 1.7M) lack corresponding bounding boxes because the speaker is off-screen or obscured. Standard "early fusion" models often fail in this environment because noisy or empty visual frames "poison" the optimization of the audio branch.

Proposed Method: Disentangled Architecture

To address the dataset's inherent noise, QuAVF moves away from joint training toward a modular, quality-aware architecture.

The QuAVF architecture features disentangled audio and visual branches with a quality-aware fusion module.

The system consists of two primary components:

  1. Audio Branch: Utilizes a Whisper-small encoder, pre-trained on 680,000 hours of speech, to extract robust semantic cues from the audio stream.
  2. Vision Branch: Employs a ResNet-50 backbone to process 15-frame sequences.

By training these branches separately, the audio branch can leverage the full dataset (including frames without visible faces), while the vision branch focuses exclusively on high-quality visual samples.

Quality-Aware Processing and Fusion

The core innovation of QuAVF is the integration of a Face Quality Score Prediction Module.

The Face Quality Score Prediction Module uses facial landmarks to assess the reliability of visual data.

Face Quality Scoring

A pre-trained facial landmark model generates a confidence score for each frame. High scores indicate clear, visible faces, while low scores suggest blur or absence. During training, this score filters out noisy visual data. During inference, the score is quantized into a one-hot vector to help the model categorize visual reliability.

Dynamic Fusion

Instead of simple concatenation, QuAVF uses a weighted sum for the final prediction: $$Final\_Score = (Quality \times Vision\_Score) + ((1 - Quality) \times Audio\_Score)$$ This ensures that when visual data is unreliable, the system defaults to the more robust audio branch.

Key Insights and Results

A critical finding was that an audio-only model initially outperformed joint audio-visual models, confirming that poor visual data hinders audio feature learning during joint optimization. Disentanglement allowed each modality to reach its potential.

QuAVF achieved 67.4% mean Average Precision (mAP) on the test set, securing first place on the Ego4D leaderboard. This represents a significant improvement over the 57.05% mAP baseline. Ablation studies confirmed that the quality-aware fusion module provides a consistent performance boost across noisy environments.

Conclusion

QuAVF demonstrates that a "data-centric" approach—prioritizing data quality and modularity over model complexity—is highly effective for real-world egocentric tasks. Future work will focus on narrowing the performance gap between validation and test sets and investigating why visual branches continue to lag behind audio performance in independent evaluations.

Summary updated

Abstract

This technical report describes our QuAVF@NTU-NVIDIA submission to the Ego4D Talking to Me (TTM) Challenge 2023. Based on the observation from the TTM task and the provided dataset, we propose to use two separate models to process the input videos and audio. By doing so, we can utilize all the labeled training data, including those without bounding box labels. Furthermore, we leverage the face quality score from a facial landmark prediction model for filtering noisy face input data. The face quality score is also employed in our proposed quality-aware fusion for integrating the results from two branches. With the simple architecture design, our model achieves 67.4% mean average precision (mAP) on the test set, which ranks first on the leaderboard and outperforms the baseline method by a large margin. Code is available at: https://github.com/hsi-che-lin/Ego4D-QuAVF-TTM-CVPR23
BibTeX
@misc{lin2023quavf,
  author = {Hsi-Che Lin and Chien-Yi Wang and Min-Hung Chen and Szu-Wei Fu and Yu-Chiang Frank Wang},
  title = {{QuAVF: Quality-aware Audio-Visual Fusion for Ego4D Talking to Me Challenge}},
  year = {2023},
  eprint = {2306.17404},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.17404},
}
Figure