AlignNet: A Unifying Approach to Audio-Visual Alignment

Jianren Wang, Zhaoyuan Fang, Hang ZhaoWACV 2020

TL;DR

This paper introduces AlignNet, a novel end-to-end deep learning model for synchronizing audio and video streams, even with complex, non-uniform temporal misalignments. By leveraging attention, feature pyramids, and a novel warping mechanism, AlignNet achieves state-of-the-art performance on dance-music and lip-speech alignment tasks, outperforming existing methods significantly.

What Problem Does This Paper Address?

Synchronizing audio and video is crucial for realistic multimedia experiences, from dance performances to film dubbing. However, real-world recordings often suffer from temporal misalignments that are not simple, constant shifts. Dancers might miss beats, or lip movements in dubbed scenes might not perfectly match the audio. These misalignments can be irregular and change over time, making them incredibly difficult to correct. Existing methods often struggle with these non-uniform distortions, either by assuming global offsets or by relying on coarse temporal resolutions. This paper aims to bridge this gap by developing a unified approach that can handle arbitrary temporal misalignments in audio-visual streams.

Methods

AlignNet is an end-to-end trainable model that learns a dense correspondence between video frames and audio segments. Its core principles include:

  • Attention Modules: Both spatial attention (highlighting important keypoints in video) and temporal attention (focusing on significant audio segments) are used to prioritize relevant information.
  • Multimodal Feature Pyramids: The model constructs multi-level feature representations for both video and audio, allowing it to capture information at different temporal scales.
  • Time Warping: A novel warping layer is introduced to adjust video features based on previously estimated correspondences. This allows the model to iteratively refine its alignment by focusing on residual misalignments.
  • Affinity Function: A correlation mechanism is used to model the similarity between video and audio features at each level of the pyramid, helping to establish correspondences.

The model takes pose keypoints (from OpenPose) and normalized log-mel spectrograms as input, processing them through convolutional layers and the aforementioned modules to predict a temporal displacement for each video frame. The training objective includes both a correspondence loss (L1 regression) and a monotonic loss to ensure temporally consistent alignments.

Datasets

The paper introduces Dance50, a new dataset for dance-music alignment. It comprises 50 hours of dancing clips from over 200 dancers, with approximately 10,000 training, 2,000 validation, and 2,000 testing videos. These clips are sourced from K-pop dance covers and annotated with 2D skeleton keypoints using OpenPose.

For lip-speech alignment, the authors use a subset of the VoxCeleb2 dataset, consisting of 22,000 video clips (73 hours total) of human speech.

Key Insights and Ideas

  • Unified Approach for Irregular Misalignment: The most significant insight is the development of a single, end-to-end model capable of handling arbitrary, non-uniform temporal misalignments, which are common in real-world scenarios.
  • Hierarchical Processing is Key: The use of feature pyramids and iterative warping allows the model to progressively refine its understanding of correspondences, from coarse to fine temporal scales.
  • Attention Guides the Alignment: Both spatial and temporal attention mechanisms prove crucial for focusing on the most informative parts of the audio and video, leading to more accurate alignments.
  • Pose Keypoints are Robust: Using pose keypoints instead of raw image frames makes the model more resilient to variations in lighting, outfits, and other visual factors.

Key Results

  • Dance-Music Alignment: AlignNet achieved a significant improvement on the Dance50 dataset, reducing the Average Frame Error (AFE) to 0.94 and increasing accuracy to 89.60%. This is a substantial gain compared to baselines like SyncNet (AFE 6.58, Accuracy 25.88) and SyncNet+DTW (AFE 4.27, Accuracy 38.29).
  • Lip-Speech Alignment: On the VoxCeleb2 subset, AlignNet also outperformed baselines, achieving an AFE of 1.03 and accuracy of 81.05% (compared to SyncNet+DTW's 3.96 AFE and 35.94% accuracy).
  • Qualitative Improvements: Visualizations show AlignNet's ability to closely recover original aligned poses and predict distortions more accurately than baselines, especially for subtle temporal changes.
  • Retargeting Success: Subjective user studies indicated that AlignNet-generated retargeted videos (both dance and speech) were preferred over direct combinations, demonstrating its practical utility.

Limitations and Future Work

  • Temporal Attention vs. Keypoint Attention: While both attention mechanisms improved performance, temporal attention showed less impact than keypoint attention. The authors hypothesize this might be because music beats are more implicitly encoded in the network. Future work could explore more sophisticated temporal attention mechanisms.
  • Dance Retargeting Performance: While good, AlignNet's performance in dance retargeting was slightly worse than direct combinations in subjective studies. This is attributed to the similar beats in many dance music pieces.
  • Generalization: The paper demonstrates generalization across dance and speech tasks, but further exploration into other audio-visual synchronization scenarios could be beneficial.
  • Real-world Data: The authors hope their work inspires further research into audio-visual synchronization for complex, real-world scenarios with irregular misalignments.

Summary updated

BibTeX
@inproceedings{wang2020alignnet,
  author = {Jianren Wang and Zhaoyuan Fang and Hang Zhao},
  title = {{AlignNet: A Unifying Approach to Audio-Visual Alignment}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure