Abstract
Detecting synchrony between audio and visual speech signals is a critical challenge for applications such as speaker identification, lip-sync correction, and biometric anti-spoofing. Traditional methods often fail because they treat speech frames as independent events and are highly sensitive to timing misalignments. This paper introduces Generalized Bimodal Linear Prediction (BLPC), a framework that models temporal correlations by predicting current audio features from a window of past and future bimodal observations. Evaluated on the CMU Audio-Visual Profile Frontal Corpus, the BLPC approach achieves a 50% relative reduction in Equal Error Rate (EER) compared to standard mutual information methods and remains robust against hardware-induced delays of up to 100ms.
Introduction
Audio-visual (AV) synchrony detection determines whether a speaker's voice matches their lip movements. This technology is fundamental in multimedia processing, helping to identify the active speaker in a crowded environment or detect if a video has been dubbed.
The core challenge in this field is that speech is a continuous, evolving signal where current states are highly correlated with previous ones. Most existing techniques, such as Mutual Information (MI) or Hypothesis Testing (HT), treat every frame of video and audio as an independent event. Furthermore, real-world data often suffers from small "jitters" or misalignments due to hardware capture delays, which cause traditional frame-by-frame methods to fail. This research addresses these issues by modeling the temporal evolution of speech signals.
Methodology
The proposed Generalized Bimodal Linear Prediction (BLPC) model predicts the current audio signal based on a window of past and future information. Unlike standard linear prediction, which is purely causal (looking only at the past), the "Generalized" variants (BLPC-2 and BLPC-3) incorporate non-causal information from future video and audio frames.
Feature Extraction and Correlation
Because audio (MFCCs) and video (geometric lip features) exist in different dimensions and scales, the researchers utilize Canonical Correlation Analysis (CCA). This projects both streams into a shared space where they are most highly correlated before the prediction model is applied.
As shown in the figure above, the visual features are derived through a multi-step process. The original image is color-thresholded to isolate the lip region, which is then refined into a final binary mask. From this mask, geometric features such as the height and width of the lip opening are extracted to serve as the visual input for the bimodal model.
Synchrony Measure
The system calculates a synchrony measure ($D_\alpha$) by determining the difference between model parameters when assuming the streams are in sync versus when they are not. A large distance indicates a high likelihood that the streams are synchronized.
Experimental Setup
The study utilized the CMU Audio-Visual Profile Frontal Corpus, consisting of high-quality recordings of speakers in a frontal head pose. The data was processed into 4-second chunks. To test the system's robustness, the researchers created "out-of-sync" data by randomly pairing audio and video chunks from different segments, resulting in approximately eight hours of asynchronous data for testing.
Key Results and Insights
The BLPC approach demonstrated clear superiority over common baselines:
- Accuracy: BLPC-1 (the causal version) achieved a 50% relative reduction in EER compared to Mutual Information and a 33% reduction compared to Hypothesis Testing.
- Robustness to Delay: When artificial delays were introduced to simulate hardware lag, the performance of MI and HT plummeted. In contrast, the non-causal BLPC-2 model remained stable, maintaining a low error rate even with misalignments up to 100ms.
- Temporal Importance: The results confirm that treating speech frames as independent is a significant limitation. Modeling the evolution of features over time captures the natural flow of speech. Interestingly, while including future visual information (BLPC-2) provided a massive boost in robustness, adding future audio information (BLPC-3) offered diminishing returns.
Conclusion
Generalized Bimodal Linear Prediction provides a robust framework for AV synchrony detection by leveraging the temporal nature of speech. By incorporating non-causal visual information, the model effectively "re-syncs" data internally, overcoming common hardware capture delays. Future work will focus on extending this model to handle non-frontal poses, moving heads, and noisier environments to better suit real-world applications.
Summary updated