Sketching With Your Voice: "Non-Phonorealistic" Rendering of Sounds via Vocal Imitation

Matthew Caren, Kartik Chandra, Joshua B. Tenenbaum, Jonathan Ragan-Kelley, Karima Ma

Abstract

We present a method for automatically producing human-like vocal imitations of sounds: the equivalent of "sketching," but for auditory rather than visual representation. Starting with a simulated model of the human vocal tract, we first try generating vocal imitations by tuning the model's control parameters to make the synthesized vocalization match the target sound in terms of perceptually-salient auditory features. Then, to better match human intuitions, we apply a cognitive theory of communication to take into account how human speakers reason strategically about their listeners. Finally, we show through several experiments and user studies that when we add this type of communicative reasoning to our method, it aligns with human intuitions better than matching auditory features alone does. This observation has broad implications for the study of depiction in computer graphics.

Introduction: The Missing Quadrant

In the field of computer graphics, researchers have long distinguished between photorealistic rendering and non-photorealistic rendering (NPR). While we have sophisticated tools for both realistic auditory simulation and realistic visual rendering, the "missing quadrant" remains abstract auditory representation—the sonic equivalent of a line drawing or a sketch.

When humans describe sounds, we frequently use vocal imitation. Interestingly, an effective imitation is rarely a literal acoustic match. For instance, when mimicking a motorboat, a human might ignore the loud, broadband noise of splashing water to focus on the low-pitched rumble of the engine. This is because the engine sound is more distinctive and communicative. This paper introduces a framework for "non-phonorealistic" sound rendering that treats vocal imitation as a strategic optimization problem rather than a simple signal-matching task.

Methodology

The researchers developed a three-stage model to bridge the gap between literal sound synthesis and human-like communicative imitation.

1. Physical Vocal Synthesis

To ensure the output remains "vocal," the system utilizes a controllable source-filter model implemented in the FAUST language. This model simulates the physical constraints of the human vocal tract. By discretizing the control space, the authors generated a library of approximately 161,000 possible human utterances, providing a finite search space for the optimization process.

2. The Feature-Matching Baseline

As a baseline, the system attempts to match a target sound (the "referent") by optimizing for 19 low-level auditory features. These include spectral centroid, flatness, and loudness. While this produces an imitation that is acoustically similar, it often fails to capture the "essence" of the sound that a human would prioritize.

3. Communicative Reasoning via RSA

To move beyond literal matching, the authors integrated the Rational Speech Acts (RSA) framework. RSA is a cognitive model that views communication as a recursive game. In this setup, a "Level-2" speaker selects an utterance by predicting how a "Level-1" listener would interpret it.

The model also incorporates:

  • Vocal Costs: Penalties for utterances that are physically difficult to produce (e.g., extremely loud or rapid sounds).
  • Semantic Utility: Using the AudioSet ontology, the model calculates utility based on category relationships. If a listener identifies a "drill" as a "jackhammer," the speaker receives partial credit because the sounds are semantically related, even if the specific label is incorrect.

Evaluation and Results

The system was evaluated using the FSD50K sound library and compared against the VocalImitationSet (a dataset of actual human imitations).

Human Alignment

One of the most significant findings was that the "full model"—which includes RSA and cost constraints—achieved a correlation of $R^2 = 0.81$ with human phonetic features. This significantly outperformed the feature-matching baseline ($R^2 \approx 0.57$). Remarkably, this alignment emerged "zero-shot"; the model was never trained on human vocal data, but rather derived its behavior from the logic of communication and the physics of the vocal tract.

User Preference and Retrieval

In a study involving 60 participants, users consistently preferred the imitations generated by the RSA-enabled model over the baseline. Furthermore, in a 3-way retrieval task, the system's internal "listener" model could correctly identify the target animal sound 58% of the time. While this is lower than the 70% accuracy achieved by humans, it is a marked improvement over the 46% accuracy of the baseline.

Discussion and Future Work

This research suggests that depiction—whether visual or auditory—is an optimization problem: finding the stimulus that best induces the desired concept in a listener's mind. By prioritizing informativeness over mere saliency, the model captures the "pragmatic strengthening" inherent in human communication.

Limitations

Despite its success, the model currently faces several hurdles:

  • Temporal Complexity: It is optimized for "sound textures" (like rain or engines) and struggles with sounds that have complex temporal structures, such as music.
  • Phonetic Constraints: The current vocal tract model lacks certain consonants (e.g., the [z] sound), which limits its ability to imitate certain insects or mechanical hums.
  • Cultural Context: The model focuses on iconicity (resemblance) and does not yet account for cultural onomatopoeia (e.g., "thump-thump" for a heartbeat).

Applications

The authors envision this technology powering new "sketch-based" interfaces for sound designers. Instead of scrolling through thousands of files, a designer could simply imitate a sound to search a library or synthesize a custom effect, using their voice as a high-level brush for auditory sketching.

Summary updated

Abstract

We present a method for automatically producing human-like vocal imitations of sounds: the equivalent of "sketching," but for auditory rather than visual representation. Starting with a simulated model of the human vocal tract, we first try generating vocal imitations by tuning the model's control parameters to make the synthesized vocalization match the target sound in terms of perceptually-salient auditory features. Then, to better match human intuitions, we apply a cognitive theory of communication to take into account how human speakers reason strategically about their listeners. Finally, we show through several experiments and user studies that when we add this type of communicative reasoning to our method, it aligns with human intuitions better than matching auditory features alone does. This observation has broad implications for the study of depiction in computer graphics.
BibTeX
@inproceedings{caren2024sketching,
  author = {Matthew Caren and Kartik Chandra and Joshua B. Tenenbaum and Jonathan Ragan-Kelley and Karima Ma},
  title = {{Sketching With Your Voice: "Non-Phonorealistic" Rendering of Sounds via Vocal Imitation}},
  booktitle = {SIGGRAPH Asia},
  year = {2024},
  doi = {10.1145/3680528.3687679},
}
Figure