TL;DR
Sketch2Sound is a lightweight adapter for text-to-audio models that allows users to control sound generation using vocal imitations or "sketches." By conditioning on interpretable signals like loudness, pitch, and brightness, it enables precise gestural control while maintaining high audio quality and semantic adherence to text prompts.
What Problem Does This Paper Address?
While text-to-audio (TTA) models have reached impressive levels of quality, they often function as "black boxes" regarding temporal control. If a sound designer needs an engine to rev at a specific moment or a bird to chirp with a particular rhythm, a text prompt like "engine revving" is far too imprecise.
Foley artists traditionally solve this by "performing" sounds in sync with visuals using their bodies and voices. Sketch2Sound aims to bring this intuitive, gestural interaction to AI-driven audio generation. The core challenge is bridging the gap between a rough vocal imitation—which has the right timing but the wrong timbre—and a high-quality, semantically accurate sound effect.
Methods
Sketch2Sound is built as a lightweight extension to a pre-trained Latent Diffusion Transformer (DiT). Instead of retraining a massive model from scratch, the authors introduce a simple but effective conditioning mechanism:
- Feature Extraction: The system extracts three time-varying control signals from an input audio "sketch" (like a user humming or beatboxing): Loudness, Spectral Centroid (perceived brightness), and Pitch Probabilities.
- Linear Projection: These signals are passed through a single trainable linear layer per control and added directly to the noisy latents used as input to the diffusion model. This makes the architecture much more efficient than methods like ControlNet, which require duplicating the entire encoder.
- Random Median Filtering: This is the paper's key innovation. During training, the authors apply median filters of varying window sizes to the control curves. This teaches the model to follow the "gist" of the sketch rather than over-fitting to the specific vocal artifacts of the human voice. At inference time, users can adjust this filter size to navigate the trade-off between strict adherence to the sketch and overall audio quality.
Datasets
- VimSketch: The primary benchmark used for evaluation, consisting of approximately 12,000 vocal imitations paired with text descriptions and ground-truth reference sounds.
- Proprietary & CC-licensed Datasets: A large mix of high-quality sound effects used for the initial pre-training of the base text-to-audio DiT model.
Key Insights and Ideas
- Sketching vs. Mimicry: The authors argue that a generative model shouldn't try to mimic a vocal imitation (which would sound like a person making noises) but should treat it as a sketch—a rough guide for the energy and pitch of the target sound.
- Implicit Semantic Mapping: One of the most striking findings is that the model learns to interpret control signals based on the text prompt's context. For example, in a "forest ambience" prompt, sudden bursts of loudness are automatically interpreted as bird chirps without the user needing to specify "birds" in the text.
- Efficiency: The entire adapter requires only 40,000 steps of fine-tuning, making it a highly practical "plug-and-play" module for existing TTA systems.
Key Results
- Precision Control: Adding control signals significantly reduces error in loudness and pitch compared to text-only baselines.
- The Precision-Quality Trade-off: The authors demonstrate that smaller median filters allow for more "precise" following of the sketch, while larger filters produce more "sketchlike" results that often have higher audio quality and better text adherence.
- Semantic Intelligence: Qualitative results show the model correctly mapping unpitched noise in a sketch to snare drums and pitched segments to bass drums when prompted with "drum beat."
Limitations and Future Work
- Centroid Entanglement: The spectral centroid (brightness) control can sometimes "leak" the room tone or background noise of the user's recording environment into the generated audio.
- Silence Handling: The model occasionally struggles when the input signal is very quiet or over-compressed, as the centroid becomes less reliable.
- Future Directions: The authors suggest exploring independent guidance scales for each control signal and expanding the set of interpretable controls to give sound artists even more granular "knobs" for their creative process.
Summary updated