Abstract
We introduce RealSync, a novel dataset designed to significantly enhance the training and evaluation of models for audio-visual synchronization (AV Sync) tasks. Sourced from high-quality YouTube channels, RealSync covers a wide range of content domains, providing improved scale, diversity, and alignment compared to existing datasets. Alongside this, we present StreamSync, a backbone-agnostic model incorporating a streaming mechanism that processes consecutive video segments dynamically to refine synchronization predictions. StreamSync outperforms existing models, offering superior accuracy with minimal computational cost.
Introduction: The Challenge of Real-World AV Sync
Audio-visual synchronization is critical for a seamless viewing experience, yet real-world video presents significant hurdles. Existing models often struggle with "sparse" events—such as a golf club strike—or voiceovers where the speaker is off-screen. Furthermore, current datasets like LRS3 are often limited to short, speech-centric clips that do not reflect the complexity of professional broadcasts. High-accuracy models also typically demand substantial computational resources, making them impractical for real-time edge deployment.
StreamSync: A Continuous Synchronization Framework
To address these issues, we propose StreamSync. Unlike traditional models that treat short video windows in isolation, StreamSync views synchronization as a continuous, temporal process.
The core of the framework is the Streaming Mechanism. It employs a "Snapshot Predictor" to generate a "Streaming Token" for each segment. These tokens are then processed by a recurrent "Streaming Head" (a Transformer or LSTM) that accumulates evidence over time. This allows the model to "smooth out" transient prediction errors and gain confidence from persistent synchronization offsets. Because it is backbone-agnostic, StreamSync can be integrated with various existing AV sync models by adding this temporal layer.
The RealSync Dataset
A major contribution of this work is RealSync, a benchmark curated specifically for professional broadcast environments. It comprises 927 hours of video from 670 high-quality YouTube sources, including news and sports.
Unlike datasets focused solely on talking heads, RealSync includes a wide variety of audio events. This diversity is crucial for training models to recognize sparse synchronization cues, such as the sound of a ball being hit, which provide definitive sync points even in the absence of visible speech.
Key Insights and Performance
Our analysis revealed that many errors in current models are uncorrelated and appear as random noise. By learning temporal patterns through the streaming head, StreamSync effectively cancels out this noise. Notably, an LSTM-based streaming head adds only 3.8% to inference costs while significantly boosting accuracy.
In evaluations on the RealSync dataset, StreamSync achieved 84.5% accuracy (within a ±1 bin tolerance), a dramatic improvement over the 38.1% achieved by standard non-streaming models. Furthermore, StreamSync is nearly 9 times faster than the state-of-the-art Synchformer (19.2ms vs 171ms per inference), demonstrating its suitability for real-time applications.
Limitations and Future Directions
While StreamSync marks a significant advancement, challenges remain. The model can struggle with subtle, continuous visual cues, such as a musician's finger movements, and complex scenes with multiple simultaneous speakers. Additionally, while the current model assumes a persistent offset, future iterations will aim to adapt to fluctuating network latencies and jitter.
For more details and code, visit the project repository.
Summary updated