Abstract
The Ego4D "Talking to Me" (TTM) challenge requires identifying if a person in an egocentric video is addressing the camera wearer. This task is complicated by significant noise in the Ego4D dataset, where many frames lack visual bounding boxes despite active vocalization. We present QuAVF, a data-centric approach that disentangles audio and visual processing. By introducing a facial-landmark-based "quality score," QuAVF intelligently filters training data and dynamically weights multi-modal fusion during inference. QuAVF achieved first place in the 2023 TTM challenge with a 67.4% mAP.
Introduction
The "Talking to Me" (TTM) task is a foundational capability for social robotics and augmented reality, requiring a system to determine social engagement from a first-person perspective. However, the Ego4D dataset presents a unique challenge: ground-truth labels are based on vocal activity, but approximately 40% of labeled frames (0.7M out of 1.7M) lack corresponding bounding boxes because the speaker is off-screen or obscured. Standard "early fusion" models often fail in this environment because noisy or empty visual frames "poison" the optimization of the audio branch.
Proposed Method: Disentangled Architecture
To address the dataset's inherent noise, QuAVF moves away from joint training toward a modular, quality-aware architecture.
The system consists of two primary components:
- Audio Branch: Utilizes a Whisper-small encoder, pre-trained on 680,000 hours of speech, to extract robust semantic cues from the audio stream.
- Vision Branch: Employs a ResNet-50 backbone to process 15-frame sequences.
By training these branches separately, the audio branch can leverage the full dataset (including frames without visible faces), while the vision branch focuses exclusively on high-quality visual samples.
Quality-Aware Processing and Fusion
The core innovation of QuAVF is the integration of a Face Quality Score Prediction Module.
Face Quality Scoring
A pre-trained facial landmark model generates a confidence score for each frame. High scores indicate clear, visible faces, while low scores suggest blur or absence. During training, this score filters out noisy visual data. During inference, the score is quantized into a one-hot vector to help the model categorize visual reliability.
Dynamic Fusion
Instead of simple concatenation, QuAVF uses a weighted sum for the final prediction: $$Final\_Score = (Quality \times Vision\_Score) + ((1 - Quality) \times Audio\_Score)$$ This ensures that when visual data is unreliable, the system defaults to the more robust audio branch.
Key Insights and Results
A critical finding was that an audio-only model initially outperformed joint audio-visual models, confirming that poor visual data hinders audio feature learning during joint optimization. Disentanglement allowed each modality to reach its potential.
QuAVF achieved 67.4% mean Average Precision (mAP) on the test set, securing first place on the Ego4D leaderboard. This represents a significant improvement over the 57.05% mAP baseline. Ablation studies confirmed that the quality-aware fusion module provides a consistent performance boost across noisy environments.
Conclusion
QuAVF demonstrates that a "data-centric" approach—prioritizing data quality and modularity over model complexity—is highly effective for real-world egocentric tasks. Future work will focus on narrowing the performance gap between validation and test sets and investigating why visual branches continue to lag behind audio performance in independent evaluations.
Summary updated