Collaborative Learning to Generate Audio-Video Jointly

Vinod K. Kurmi, Vipul Bajaj, Badri N. Patro, K. S. Venkatesh, Vinay P. Namboodiri, Preethi Jyothi

Introduction

Generative Adversarial Networks (GANs) have achieved remarkable success in synthesizing high-fidelity images, videos, and audio. However, most existing approaches focus on a single modality. When multi-modal generation is attempted, it typically takes the form of cross-modal translation—generating video from audio or vice versa. This paper introduces a novel collaborative learning framework designed for joint generation, where both audio and video are synthesized simultaneously from a single shared latent noise vector. This approach aims to capture the inherent physical correlations between sight and sound, such as the rhythmic motion of a musician and the resulting acoustic waveform.

Methodology

The proposed framework leverages the synergy between two specialized generation modules, synchronized through a shared latent space and a sophisticated discriminator system.

Overview of the joint audio-video generation framework.

1. Audio Generation Module

The audio component is based on the WaveGAN architecture, utilizing 1-D deconvolutional layers to generate raw audio waveforms. A critical challenge in raw audio GANs is the presence of "checkerboard" artifacts. To mitigate this, the authors employ a "phase shuffle" technique, which prevents the discriminator from relying on trivial phase alignments, forcing it to learn more robust acoustic features.

2. Video Generation Module

The video module is inspired by MoCoGAN, which decomposes video into content (static scene elements) and motion (temporal dynamics). By generating a sequence of frames that maintain temporal consistency, the model ensures that the visual output is both realistic and fluid.

3. Joint Discriminators

The "secret sauce" of this framework is the joint discriminator module. Unlike standard GANs that use independent discriminators for each modality, this system uses a pair of joint discriminators. These are trained to distinguish between real audio-video pairs and various "fake" combinations (e.g., generated audio with real video). This collaborative training forces the generators to produce outputs that are not only individually realistic but also perfectly synchronized.

Datasets

To evaluate the framework, the researchers curated two specific benchmarks where audio and video are tightly coupled:

  • Sub-URMP-Video: A subset of the University of Rochester Multi-Modal Music Performance (URMP) dataset. It includes 13 instrument categories, processed into 1-second clips where 10 frames of video are paired with 1 second of audio.
  • Hand Motion Music Dataset (HMMD): A new dataset featuring over 6,000 pairs from YouTube clips of instruments like the Tabla and Xylophone. These were chosen specifically because the sound is a direct physical consequence of the visible hand motions.

Results and Discussion

The "Joint Model" (JM) was compared against a "Baseline" (BL) where audio and video were generated independently. The results consistently favored the collaborative approach across both quantitative and qualitative metrics.

Quantitative comparison of human evaluation and Inception Scores.

As shown in the table above, the Joint Model achieved higher Inception Scores (IS) and significantly better human preference ratings across both datasets. For the HMMD dataset, human evaluators rated the Joint Model at 3.5 out of 5, compared to 2.9 for the baseline. Additionally, audio quality improved significantly; on the Sub-URMP dataset, the Frechet Inception Distance (FID) dropped from 111.25 (Baseline) to 97.43 (Joint Model), indicating higher acoustic fidelity.

Conclusion

This research demonstrates that treating audio and video generation as a collaborative task rather than independent problems leads to superior synchronization and realism. By anchoring both modalities in a shared latent space, the model learns the underlying physical laws that govern multi-modal phenomena. While current limitations include short clip lengths and the complexity of non-periodic motions, this collaborative paradigm serves as a catalyst for more advanced multi-modal synthesis in the future.

Summary updated

Abstract

There have been a number of techniques that have demonstrated the generation of multimedia data for one modality at a time using GANs, such as the ability to generate images, videos, and audio. However, so far, the task of multi-modal generation of data, specifically for audio and videos both, has not been sufficiently well-explored. Towards this, we propose a method that demonstrates that we are able to generate naturalistic samples of video and audio data by the joint correlated generation of audio and video modalities. The proposed method uses multiple discriminators to ensure that the audio, video, and the joint output are also indistinguishable from real-world samples. We present a dataset for this task and show that we are able to generate realistic samples. This method is validated using various standard metrics such as Inception Score, Frechet Inception Distance (FID) and through human evaluation.
BibTeX
@inproceedings{kurmi2021collaborative,
  author = {Vinod K. Kurmi and Vipul Bajaj and Badri N. Patro and K. S. Venkatesh and Vinay P. Namboodiri and Preethi Jyothi},
  title = {{Collaborative Learning to Generate Audio-Video Jointly}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9413802},
}
Figure