Abstract
VarietySound introduces a novel framework for Timbre-Controllable Video to Sound Generation (TCVSG). While existing video-to-sound (V2S) methods can generate audio synchronized with visual events, they often lack control over the specific "texture" or timbre of the sound. VarietySound allows users to provide a reference audio clip to specify the desired timbre while using a silent video to dictate the timing. By employing unsupervised information disentanglement, the model separates temporal cues, acoustic characteristics, and background noise. Evaluations on the VAS dataset show that VarietySound achieves superior performance in subjective realism, temporal alignment, and timbre similarity compared to baseline methods.
Introduction
In professional video production, the specific quality of a sound effect—its timbre—is as important as its timing. For instance, a "glass shattering" sound might need to be sharp and crystalline or dull and heavy depending on the scene's context. Traditional V2S models typically produce an "average" sound for a given visual event, offering no stylistic control.
To address this, we propose the task of Timbre-Controllable Video to Sound Generation (TCVSG). As illustrated in Figure 1, while existing tasks focus solely on generating audio $\hat{A}$ from video $V$, TCVSG incorporates a reference audio $A$ to guide the acoustic style of the output.
Methodology
The VarietySound framework is built upon the principle of Unsupervised Information Disentanglement. The model decomposes the target audio into three distinct components:
- Temporal Information: Extracted from video frames $V$ via a Temporal Encoder (TE). This provides the "when" of the sound.
- Acoustic Information: Extracted from reference audio $A$ via an Acoustic Encoder (AE). To ensure the model learns only timbre and not timing from the reference, a Random Resampling (RR) operation shuffles and warps the audio's time structure.
- Background Information: Extracted via a Background Encoder (BE) using Energy Masking (EM) to isolate low-energy ambient noise.
These components are fused by a Mel Decoder to reconstruct the final mel-spectrogram. The system is trained within a Generative Adversarial Network (GAN) framework using two specialized discriminators: a Multi-window Mel Discriminator for overall realism and a Time-domain Alignment Discriminator (TDAD) to ensure precise synchronization between visual peaks and audio events.
Experiments and Results
The model was evaluated on the VAS (Video-to-Audio Synthesis) dataset, comprising 12,541 videos across eight categories (e.g., Baby, Dog, Drum, Gun). VarietySound was compared against a cascade baseline combining RegNet and SpeechSplit.
- Subjective Quality: VarietySound achieved a Mean Opinion Score (MOS) of 4.04, significantly higher than the baseline's 2.99.
- Temporal Alignment: The model reached a score of 4.23, nearing the ground truth of 4.41, validating the effectiveness of the TDAD.
- Timbre Fidelity: Using cosine similarity of acoustic features, VarietySound scored 0.90 (vs. 0.84 for the baseline), demonstrating its ability to faithfully replicate the reference audio's texture.
Conclusion
VarietySound successfully enables controllable sound generation for silent videos by disentangling timing from timbre. The use of Random Resampling and dedicated background modeling allows for high-fidelity synthesis that respects both visual cues and stylistic preferences. Future work will explore cross-category timbre transfer and direct waveform generation to further enhance the versatility of the framework.
Summary updated