Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations

Hugo Flores García, Oriol Nieto, Justin Salamon, Bryan Pardo, Prem Seetharaman

TL;DR

Sketch2Sound is a lightweight adapter for text-to-audio models that allows users to control sound generation using vocal imitations or "sketches." By conditioning on interpretable signals like loudness, pitch, and brightness, it enables precise gestural control while maintaining high audio quality and semantic adherence to text prompts.

What Problem Does This Paper Address?

While text-to-audio (TTA) models have reached impressive levels of quality, they often function as "black boxes" regarding temporal control. If a sound designer needs an engine to rev at a specific moment or a bird to chirp with a particular rhythm, a text prompt like "engine revving" is far too imprecise.

Foley artists traditionally solve this by "performing" sounds in sync with visuals using their bodies and voices. Sketch2Sound aims to bring this intuitive, gestural interaction to AI-driven audio generation. The core challenge is bridging the gap between a rough vocal imitation—which has the right timing but the wrong timbre—and a high-quality, semantically accurate sound effect.

Methods

Sketch2Sound is built as a lightweight extension to a pre-trained Latent Diffusion Transformer (DiT). Instead of retraining a massive model from scratch, the authors introduce a simple but effective conditioning mechanism:

  1. Feature Extraction: The system extracts three time-varying control signals from an input audio "sketch" (like a user humming or beatboxing): Loudness, Spectral Centroid (perceived brightness), and Pitch Probabilities.
  2. Linear Projection: These signals are passed through a single trainable linear layer per control and added directly to the noisy latents used as input to the diffusion model. This makes the architecture much more efficient than methods like ControlNet, which require duplicating the entire encoder.
  3. Random Median Filtering: This is the paper's key innovation. During training, the authors apply median filters of varying window sizes to the control curves. This teaches the model to follow the "gist" of the sketch rather than over-fitting to the specific vocal artifacts of the human voice. At inference time, users can adjust this filter size to navigate the trade-off between strict adherence to the sketch and overall audio quality.
Overview of the Sketch2Sound architecture, showing how control signals are extracted from a vocal imitation and fed into the DiT.

Datasets

  • VimSketch: The primary benchmark used for evaluation, consisting of approximately 12,000 vocal imitations paired with text descriptions and ground-truth reference sounds.
  • Proprietary & CC-licensed Datasets: A large mix of high-quality sound effects used for the initial pre-training of the base text-to-audio DiT model.

Key Insights and Ideas

  • Sketching vs. Mimicry: The authors argue that a generative model shouldn't try to mimic a vocal imitation (which would sound like a person making noises) but should treat it as a sketch—a rough guide for the energy and pitch of the target sound.
  • Implicit Semantic Mapping: One of the most striking findings is that the model learns to interpret control signals based on the text prompt's context. For example, in a "forest ambience" prompt, sudden bursts of loudness are automatically interpreted as bird chirps without the user needing to specify "birds" in the text.
  • Efficiency: The entire adapter requires only 40,000 steps of fine-tuning, making it a highly practical "plug-and-play" module for existing TTA systems.

Key Results

  • Precision Control: Adding control signals significantly reduces error in loudness and pitch compared to text-only baselines.
  • The Precision-Quality Trade-off: The authors demonstrate that smaller median filters allow for more "precise" following of the sketch, while larger filters produce more "sketchlike" results that often have higher audio quality and better text adherence.
The trade-off between precision and quality: smaller filters follow the sketch more closely, while larger filters improve audio quality and text adherence.
  • Semantic Intelligence: Qualitative results show the model correctly mapping unpitched noise in a sketch to snare drums and pitched segments to bass drums when prompted with "drum beat."
Fig. 3: (left) When prompted with “forest ambience”, bursts of loudness in the controls become of birds without prompting the model to do so. (right) With “bass drum, snare drum”, the model places snares in unpitched areas and bass drums in pitched areas.

Limitations and Future Work

  • Centroid Entanglement: The spectral centroid (brightness) control can sometimes "leak" the room tone or background noise of the user's recording environment into the generated audio.
  • Silence Handling: The model occasionally struggles when the input signal is very quiet or over-compressed, as the centroid becomes less reliable.
  • Future Directions: The authors suggest exploring independent guidance scales for each control signal and expanding the set of interpretable controls to give sound artists even more granular "knobs" for their creative process.

Summary updated

Abstract

We present Sketch2Sound, a generative audio model capable of creating high-quality sounds from a set of interpretable time-varying control signals: loudness, brightness, and pitch, as well as text prompts. Sketch2Sound can synthesize arbitrary sounds from sonic imitations (i.e.,~a vocal imitation or a reference sound-shape). Sketch2Sound can be implemented on top of any text-to-audio latent diffusion transformer (DiT), and requires only 40k steps of fine-tuning and a single linear layer per control, making it more lightweight than existing methods like ControlNet. To synthesize from sketchlike sonic imitations, we propose applying random median filters to the control signals during training, allowing Sketch2Sound to be prompted using controls with flexible levels of temporal specificity. We show that Sketch2Sound can synthesize sounds that follow the gist of input controls from a vocal imitation while retaining the adherence to an input text prompt and audio quality compared to a text-only baseline. Sketch2Sound allows sound artists to create sounds with the semantic flexibility of text prompts and the expressivity and precision of a sonic gesture or vocal imitation. Sound examples are available at https://hugofloresgarcia.art/sketch2sound/.
BibTeX
@inproceedings{garcia2025sketch,
  author = {Hugo Flores García and Oriol Nieto and Justin Salamon and Bryan Pardo and Prem Seetharaman},
  title = {{Sketch2Sound: Controllable Audio Generation via Time-Varying Signals and Sonic Imitations}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888184},
}
Figure