Abstract
While text-to-image editing has achieved significant milestones, text often lacks the expressive capacity to convey atmospheric nuances and continuous intensity found in auditory signals. This paper introduces SoundBrush, a novel framework that utilizes audio as a control signal for precise 2D image and 3D scene editing. By mapping audio features into the textual embedding space of Latent Diffusion Models (LDMs), SoundBrush enables the insertion of sounding objects and the modification of environmental scenery while maintaining structural integrity. Our approach leverages a specialized dataset of over 83,000 image-audio pairs and demonstrates superior performance in both quantitative metrics and human preference studies.
1. Introduction
The field of generative AI has largely focused on text-to-image (T2I) synthesis, yet text is inherently discrete and often fails to capture the rich, continuous variations of the physical world. For instance, the prompt "raining" provides a binary instruction, whereas an audio clip of rain can distinguish between a light drizzle and a torrential downpour.
Existing sound-guided editing methods frequently struggle with three core challenges: the tendency to generate new images rather than edit existing ones, a focus on global style over local content insertion, and the loss of structural integrity in the source image. SoundBrush addresses these limitations by bridging the gap between audio perception and visual generation, treating sound as a "brush" for localized, semantically aware editing.
2. Methodology
SoundBrush is built upon the InstructPix2Pix architecture, adapted to process audio tokens instead of, or alongside, text. The framework consists of several key components designed to align the auditory and visual domains.
2.1 Audio Mapping Network
To integrate audio into a pre-trained LDM, the system employs a CLAP (Contrastive Language-Audio Pretraining) encoder to extract high-level features from raw audio. These features are processed by a specialized mapping network that converts them into a sequence of "audio tokens." Crucially, these tokens are projected into the same embedding space as the model's textual tokens, allowing the LDM to interpret sound through its existing linguistic priors.
2.2 Joint Optimization and Loss Functions
The model is fine-tuned using LoRA (Low-Rank Adaptation), which allows for efficient updates to the LDM weights without retraining the entire network. The training process is guided by a tripartite loss function:
- Diffusion Loss ($L_{LDM}$): The standard objective for maintaining generative quality.
- Contrastive Loss (InfoNCE): Used to align the generated audio tokens with corresponding visual features, ensuring semantic consistency.
- $\ell_1$ Regularization: Applied to the token distribution to ensure stability and prevent overfitting to specific audio frequencies.
2.3 3D Scene Editing
Beyond 2D images, SoundBrush extends to 3D environments by integrating with the InstructNeRF2NeRF pipeline. This allows users to edit Neural Radiance Fields (NeRFs) using sound, ensuring that the modifications (such as adding a sounding object) remain consistent across multiple camera viewpoints.
3. Dataset Construction
A significant hurdle in audio-visual editing is the lack of paired "before-and-after" datasets. SoundBrush overcomes this through a dual-strategy data generation approach, resulting in 83,614 training pairs:
- Fully Synthetic Subset (27,056 pairs): Using GPT-4, the authors generated source and target text descriptions based on sound categories. These descriptions were then used with Prompt-to-Prompt to create visually consistent image pairs representing an edited state.
- Real-Data Subset (56,558 pairs): Leveraging the VGGSound dataset, the authors identified "sounding objects" within real video frames. They used LaMa inpainting to remove these objects, creating a "before" image. The original frame served as the "after" image, providing a ground truth for how an object should be inserted into a real-world context based on its sound.
4. Key Insights and Results
One of the most significant findings of SoundBrush is the model's ability to treat sound as a continuous variable. Because the audio signal is processed as a continuous embedding, the model naturally reflects intensity: louder sounds of hail result in more ice on the ground, and more aggressive volcanic rumbles produce denser smoke and fire. This level of granular control is difficult to replicate with discrete text prompts.
4.1 Quantitative Performance
SoundBrush was evaluated against competitive baselines, including AudioToken, GlueGen, and InstructAny2Pix. It achieved state-of-the-art results in:
- Audio-Visual Similarity (AVS): 0.261 (indicating high semantic alignment with the audio).
- Image-Image Similarity (IIS): 0.772 (demonstrating superior preservation of the original image structure).
4.2 User Preference
In a user study involving 47 participants, SoundBrush was consistently preferred over existing methods. Participants rated the model higher in three specific categories: the accuracy of audio reflection, the preservation of the original scene's structure, and overall generative quality.
5. Limitations and Future Work
Despite its successes, SoundBrush faces challenges regarding spatial logic. The model occasionally places objects in unrealistic locations—such as a train in the sky—because it lacks explicit spatial instructions, relying entirely on the audio's semantic content. Additionally, the generation of human figures is currently less refined than other objects, likely due to the lower resolution and motion blur present in the VGGSound training data.
Future research will explore the use of multi-channel (spatial) audio. By incorporating directional audio data, the model could potentially learn to infer the correct spatial placement of objects within a 2D or 3D scene, resolving current logic discrepancies.
6. Conclusion
SoundBrush represents a significant step forward in multi-modal editing. By successfully mapping audio into the textual space of diffusion models and utilizing innovative inpainting-based training, the framework provides a powerful new tool for creators to manipulate visual scenes with the atmospheric richness of sound.
Summary updated