Abstract
Humans possess the innate ability to imagine a visual scene upon hearing a sound. This research explores enabling machines with a similar capability using conditional generative adversarial networks (GANs). By integrating advanced techniques such as spectral normalization, projection discriminators, and auxiliary classifiers, the proposed model generates higher-quality images compared to naive conditional GANs. Subjective evaluations show that nearly 75% of participants find the generated images correlate well with the input audio. Furthermore, the model demonstrates an understanding of the relationship between audio volume and visual scale, adjusting the intensity of generated scenes based on input loudness.
Introduction
While significant progress has been made in text-to-image and class-to-image synthesis, audio-to-scene synthesis remains a formidable challenge. The primary obstacle is the inherent "noise" in natural audiovisual data. In many real-world videos, the audio and visual components are only loosely correlated—for instance, a recording might capture the sound of a boat engine while the camera focuses on a passenger rather than the boat itself. This discrepancy makes it difficult for models to learn a direct mapping between specific sounds and their corresponding visual representations.
Methodology
The researchers developed an audio-conditioned GAN architecture designed for stability and high-fidelity output. The system comprises several key components:
- Feature Extraction via SoundNet: Raw audio is processed through SoundNet, a deep convolutional model pretrained on massive video datasets, to produce a high-level 256-dimensional feature vector.
- Generator: A deconvolutional network takes the audio vector combined with a noise vector ($z$) to synthesize a $64 \times 64$ image.
- Projection Discriminator: Rather than simple concatenation, this component uses an inner product between audio features and intermediate discriminator features, allowing for a more nuanced judgment of the audio-visual match.
- Auxiliary Classifier: This ensures category consistency by penalizing the generator if the output image does not clearly represent the intended class (e.g., ensuring a "dog" sound results in a recognizable "dog" image).
To stabilize training, the team employed Spectral Normalization and Hinge Loss, preventing common GAN issues like mode collapse.
Data Curation
To address the correlation issues in the SoundNet Dataset, the researchers implemented an automated cleaning pipeline. They cross-referenced labels from an Inception image classifier and a SoundNet audio classifier. Pairs were only retained if both classifiers agreed on the category. This rigorous process resulted in discarding 78% of the initial data, yielding a high-quality dataset of 10,701 pairs across nine distinct categories, including planes, speedboats, and various musical instruments.
Results and Discussion
The model achieved an Inception Score (IS) of 2.83, a marked improvement over the 2.21 score of a baseline conditional GAN. Human trials further validated these results, with 73% of participants agreeing that the generated images successfully matched the input sounds.
The Volume-Scale Relationship
A significant finding was the model's ability to map audio volume to visual intensity. When the volume of a "speedboat" audio clip was increased, the generator produced images with larger splashes and more prominent wakes. This suggests the model learned that higher volume often correlates with proximity or increased physical intensity.
Semantic Understanding
The choice of audio representation proved critical. SoundNet features significantly outperformed traditional signal processing methods like MFCCs or Spectrograms. This is because SoundNet captures high-level semantic concepts rather than just raw frequency data. However, the model still faces challenges with "visually noisy" classes—such as dogs or drums—where the subject can appear in highly varied environments, unlike the more consistent backgrounds found in "plane" or "speedboat" scenes.
Conclusion
This work demonstrates the feasibility of synthesizing recognizable scenes from raw audio using a refined GAN framework. While current outputs are limited to $64 \times 64$ resolution and high-variance classes remain difficult to synthesize, the integration of projection discriminators and auxiliary classifiers provides a strong foundation. Future research will focus on increasing resolution, improving data efficiency through dual learning strategies, and further tightening the cross-modal alignment between sound and sight.
Summary updated