Sketching With Your Voice: "Non-Phonorealistic" Rendering of Sounds via Vocal Imitation

Matthew Caren, Kartik Chandra, Joshua B. Tenenbaum, Jonathan Ragan-Kelley, Karima Ma

TL;DR

This paper introduces "non-phonorealistic" sound rendering, treating vocal imitation as the auditory equivalent of sketching. By combining a physical model of the human vocal tract with a cognitive theory of communication (Rational Speech Acts), the authors developed a system that generates human-like sound imitations that prioritize communicative distinctiveness over literal acoustic matching.

What Problem Does This Paper Address?

In computer graphics, we have well-established methods for both realistic and abstract visual representation (e.g., physically-based rendering vs. line sketching). We also have realistic auditory simulation. However, the "missing quadrant"—abstract auditory representation—remains largely unexplored.

When humans try to describe a sound, we often resort to vocal imitation. Interestingly, an effective imitation doesn't necessarily sound "exactly" like the target. For example, when imitating a motorboat, humans often ignore the loud, salient splashing noise (broadband noise) and instead mimic the low-pitched engine rumble because it is more distinctive. Existing automated methods for sound synthesis usually rely on direct feature matching, which fails to capture this human intuition for abstraction and communication.

Methods

The authors propose a three-stage model built from first principles:

  1. Vocal Tract Synthesis: They use a controllable "source-filter" model (implemented in the FAUST language) that simulates the physics of the human voice. They discretized the control space to create a library of ~161,000 possible human utterances.
  2. The Baseline (Feature Matching): Initially, the system tries to match a target sound (the "referent") by optimizing for 19 low-level auditory features (e.g., spectral centroid, flatness, loudness).
  3. The RSA Module (Communicative Reasoning): To move beyond literal matching, they apply the Rational Speech Acts (RSA) framework. This cognitive model treats communication as a recursive game: a "Level-2" speaker chooses an utterance by predicting how a "Level-1" listener would interpret it.
  4. Costs and Utilities: The model incorporates a "cost" for difficult vocalizations (e.g., very loud or rapid sounds) and a "utility" based on semantic similarity. If the listener guesses a "drill" instead of a "jackhammer," the speaker is partially satisfied because they are semantically related in the AudioSet ontology.

Datasets

  • FSD50K: A library of over 51,000 audio clips used as the source for "referent" sounds.
  • VocalImitationSet: A dataset of human-generated vocal imitations used to evaluate how well the model's outputs align with actual human behavior.
  • AudioSet Ontology: A hierarchical classification of sounds used to calculate semantic "utility" (e.g., knowing that a "meow" and a "purr" both belong to the "Cat" category).

Key Insights and Ideas

  • Informative > Salient: Humans don't imitate the loudest part of a sound; they imitate the part that best distinguishes that sound from others. The RSA framework mathematically captures this "pragmatic strengthening."
  • Depiction as Optimization: Following the work of Frédo Durand, the authors view depiction (whether a sketch or a vocalization) as an optimization problem: find the stimulus that best induces the desired percept in the listener's mind.
  • Zero-Shot Alignment: Remarkably, the model was never trained on human vocal data. Its high correlation with human behavior emerges purely from modeling the constraints of the vocal tract and the logic of human communication.

Key Results

  • Human Alignment: The "full model" (including RSA and costs) achieved a correlation of $R^2 = 0.81$ with human phonetic features, significantly outperforming the feature-matching baseline ($R^2 \approx 0.57$).
  • User Preference: In a 60-person user study, participants consistently preferred the imitations generated by the full model over the baseline. In some cases, users even preferred the model's synthetic imitations over actual human-generated ones.
  • Retrieval Task: The system's "listener" model could identify the correct animal sound from a vocal imitation with 58% accuracy in a 3-way task (compared to 70% for humans and 46% for the baseline).

Limitations and Future Work

  • Temporal Structure: The current method focuses on "sound textures" (stable sounds like rain or engines) and struggles with complex temporal structures like speech or melodic music.
  • Phonetic Range: The vocal tract model lacks certain consonants (like the buzzing [z]), leading to divergences from human imitations of insects.
  • Cultural Conventions: The model operates on "iconicity" (direct resemblance) and cannot currently predict culturally evolved onomatopoeia (like saying "ba-boom" for a heartbeat, which is acoustically a poor match but a strong cultural convention).
  • Future Applications: The authors suggest this could lead to "sketch-based" interfaces for sound designers, allowing them to search sound libraries or synthesize effects using only their voice.

Summary updated

Abstract

We present a method for automatically producing human-like vocal imitations of sounds: the equivalent of "sketching," but for auditory rather than visual representation. Starting with a simulated model of the human vocal tract, we first try generating vocal imitations by tuning the model's control parameters to make the synthesized vocalization match the target sound in terms of perceptually-salient auditory features. Then, to better match human intuitions, we apply a cognitive theory of communication to take into account how human speakers reason strategically about their listeners. Finally, we show through several experiments and user studies that when we add this type of communicative reasoning to our method, it aligns with human intuitions better than matching auditory features alone does. This observation has broad implications for the study of depiction in computer graphics.
BibTeX
@inproceedings{caren2024sketching,
  author = {Matthew Caren and Kartik Chandra and Joshua B. Tenenbaum and Jonathan Ragan-Kelley and Karima Ma},
  title = {{Sketching With Your Voice: "Non-Phonorealistic" Rendering of Sounds via Vocal Imitation}},
  booktitle = {SIGGRAPH Asia},
  year = {2024},
  doi = {10.1145/3680528.3687679},
}

Loading related content...

Figure