Temporally Streaming Audio-Visual Synchronization for Real-World Videos

Jordan Voas, Wei-Cheng Tseng, Layne Berry, Xixi Hu, Puyuan Peng, James Stuedemann, David HarwathWACV 2025 · OpenReview

Abstract

We introduce RealSync, a novel dataset designed to significantly enhance the training and evaluation of models for audio-visual synchronization (AV Sync) tasks. Sourced from high-quality YouTube channels, RealSync covers a wide range of content domains, providing improved scale, diversity, and alignment compared to existing datasets. Alongside this, we present StreamSync, a backbone-agnostic model incorporating a streaming mechanism that processes consecutive video segments dynamically to refine synchronization predictions. StreamSync outperforms existing models, offering superior accuracy with minimal computational cost.

Introduction: The Challenge of Real-World AV Sync

Audio-visual synchronization is critical for a seamless viewing experience, yet real-world video presents significant hurdles. Existing models often struggle with "sparse" events—such as a golf club strike—or voiceovers where the speaker is off-screen. Furthermore, current datasets like LRS3 are often limited to short, speech-centric clips that do not reflect the complexity of professional broadcasts. High-accuracy models also typically demand substantial computational resources, making them impractical for real-time edge deployment.

StreamSync: A Continuous Synchronization Framework

To address these issues, we propose StreamSync. Unlike traditional models that treat short video windows in isolation, StreamSync views synchronization as a continuous, temporal process.

The StreamSync architecture utilizes a streaming transformer to accumulate information from consecutive video windows via a Snapshot Predictor.

The core of the framework is the Streaming Mechanism. It employs a "Snapshot Predictor" to generate a "Streaming Token" for each segment. These tokens are then processed by a recurrent "Streaming Head" (a Transformer or LSTM) that accumulates evidence over time. This allows the model to "smooth out" transient prediction errors and gain confidence from persistent synchronization offsets. Because it is backbone-agnostic, StreamSync can be integrated with various existing AV sync models by adding this temporal layer.

The RealSync Dataset

A major contribution of this work is RealSync, a benchmark curated specifically for professional broadcast environments. It comprises 927 hours of video from 670 high-quality YouTube sources, including news and sports.

RealSync features a diverse range of audio-visual occurrences beyond simple speech, including music, applause, and various environmental sounds.

Unlike datasets focused solely on talking heads, RealSync includes a wide variety of audio events. This diversity is crucial for training models to recognize sparse synchronization cues, such as the sound of a ball being hit, which provide definitive sync points even in the absence of visible speech.

Key Insights and Performance

Our analysis revealed that many errors in current models are uncorrelated and appear as random noise. By learning temporal patterns through the streaming head, StreamSync effectively cancels out this noise. Notably, an LSTM-based streaming head adds only 3.8% to inference costs while significantly boosting accuracy.

In evaluations on the RealSync dataset, StreamSync achieved 84.5% accuracy (within a ±1 bin tolerance), a dramatic improvement over the 38.1% achieved by standard non-streaming models. Furthermore, StreamSync is nearly 9 times faster than the state-of-the-art Synchformer (19.2ms vs 171ms per inference), demonstrating its suitability for real-time applications.

Limitations and Future Directions

While StreamSync marks a significant advancement, challenges remain. The model can struggle with subtle, continuous visual cues, such as a musician's finger movements, and complex scenes with multiple simultaneous speakers. Additionally, while the current model assumes a persistent offset, future iterations will aim to adapt to fluctuating network latencies and jitter.

For more details and code, visit the project repository.

Summary updated

Abstract

We introduce RealSync, a novel dataset designed to significantly enhance the training and evaluation of models for audio-visual synchronization (AV Sync) tasks. Sourced from high-quality YouTube channels, RealSync covers a wide range of content domains, providing an improved scale, diversity, and alignment with broadcast content compared to existing datasets. It features extended-length video samples, catering to the critical need for more comprehensive, real-world training and evaluation materials. Alongside this dataset, we present StreamSync, a model tailored for real-world AV Sync applications. StreamSync is designed to be backbone agnostic and incorporates a streaming mechanism that processes consecutive video segments dynamically, iteratively refining synchronization predictions. This innovative approach enables StreamSync to outperform existing models, offering superior synchronization accuracy with minimal computational cost per iteration. Together, our dataset and the StreamSync model establish a new benchmark for AVSync research, promising to drive the development of more robust and practical AVSync methods. https://github.com/jvoas655/StreamSync
BibTeX
@inproceedings{voas2025temporally,
  author = {Jordan Voas and Wei-Cheng Tseng and Layne Berry and Xixi Hu and Puyuan Peng and James Stuedemann and David Harwath},
  title = {{Temporally Streaming Audio-Visual Synchronization for Real-World Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  url = {https://openreview.net/forum?id=vdDQsSENpI},
}
Figure