VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement

Chenye Cui, Yi Ren, Jinglin Liu, Rongjie Huang, Zhou Zhao

Abstract

VarietySound introduces a novel framework for Timbre-Controllable Video to Sound Generation (TCVSG). While existing video-to-sound (V2S) methods can generate audio synchronized with visual events, they often lack control over the specific "texture" or timbre of the sound. VarietySound allows users to provide a reference audio clip to specify the desired timbre while using a silent video to dictate the timing. By employing unsupervised information disentanglement, the model separates temporal cues, acoustic characteristics, and background noise. Evaluations on the VAS dataset show that VarietySound achieves superior performance in subjective realism, temporal alignment, and timbre similarity compared to baseline methods.

Introduction

In professional video production, the specific quality of a sound effect—its timbre—is as important as its timing. For instance, a "glass shattering" sound might need to be sharp and crystalline or dull and heavy depending on the scene's context. Traditional V2S models typically produce an "average" sound for a given visual event, offering no stylistic control.

To address this, we propose the task of Timbre-Controllable Video to Sound Generation (TCVSG). As illustrated in Figure 1, while existing tasks focus solely on generating audio $\hat{A}$ from video $V$, TCVSG incorporates a reference audio $A$ to guide the acoustic style of the output.

Comparison of existing V2S tasks versus the proposed TCVSG task.

Methodology

The VarietySound framework is built upon the principle of Unsupervised Information Disentanglement. The model decomposes the target audio into three distinct components:

  1. Temporal Information: Extracted from video frames $V$ via a Temporal Encoder (TE). This provides the "when" of the sound.
  2. Acoustic Information: Extracted from reference audio $A$ via an Acoustic Encoder (AE). To ensure the model learns only timbre and not timing from the reference, a Random Resampling (RR) operation shuffles and warps the audio's time structure.
  3. Background Information: Extracted via a Background Encoder (BE) using Energy Masking (EM) to isolate low-energy ambient noise.

These components are fused by a Mel Decoder to reconstruct the final mel-spectrogram. The system is trained within a Generative Adversarial Network (GAN) framework using two specialized discriminators: a Multi-window Mel Discriminator for overall realism and a Time-domain Alignment Discriminator (TDAD) to ensure precise synchronization between visual peaks and audio events.

The VarietySound framework showing information disentanglement during training and inference.

Experiments and Results

The model was evaluated on the VAS (Video-to-Audio Synthesis) dataset, comprising 12,541 videos across eight categories (e.g., Baby, Dog, Drum, Gun). VarietySound was compared against a cascade baseline combining RegNet and SpeechSplit.

  • Subjective Quality: VarietySound achieved a Mean Opinion Score (MOS) of 4.04, significantly higher than the baseline's 2.99.
  • Temporal Alignment: The model reached a score of 4.23, nearing the ground truth of 4.41, validating the effectiveness of the TDAD.
  • Timbre Fidelity: Using cosine similarity of acoustic features, VarietySound scored 0.90 (vs. 0.84 for the baseline), demonstrating its ability to faithfully replicate the reference audio's texture.

Conclusion

VarietySound successfully enables controllable sound generation for silent videos by disentangling timing from timbre. The use of Random Resampling and dedicated background modeling allows for high-fidelity synthesis that respects both visual cues and stylistic preferences. Future work will explore cross-category timbre transfer and direct waveform generation to further enhance the versatility of the framework.

Summary updated

Abstract

Video-to-sound generation aims to generate realistic and natural sound given a video input. However, previous video-to-sound generation methods can only generate a random or average timbre without any controls of the generated sound timbre, leading to the problem that people cannot obtain the desired timbre under these methods sometimes. In this paper, we propose the task of generating sound with a specific timbre given a silent video input and a reference audio sample. To solve this task, we first use three encoders to disentangle each target sound audio into temporal, acoustic, and background information respectively, then we use a decoder to reconstruct the audio given these disentangled representations. To make the generated result achieve better quality and temporal alignment, we also adopt a mel discriminator and a temporal discriminator for the adversarial training. Our experimental results on the VAS dataset demonstrate that our method can generate high-quality audio samples with good synchronization with events in video and high timbre similarity with the reference audio. Our demos have been published on https://conferencedemos.github.io/icassp23/.
BibTeX
@inproceedings{cui2023varietysound,
  author = {Chenye Cui and Yi Ren and Jinglin Liu and Rongjie Huang and Zhou Zhao},
  title = {{VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglement}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  doi = {10.1109/icassp49357.2023.10096353},
}
Figure