TL;DR
This paper introduces "non-phonorealistic" sound rendering, treating vocal imitation as the auditory equivalent of sketching. By combining a physical model of the human vocal tract with a cognitive theory of communication (Rational Speech Acts), the authors developed a system that generates human-like sound imitations that prioritize communicative distinctiveness over literal acoustic matching.
What Problem Does This Paper Address?
In computer graphics, we have well-established methods for both realistic and abstract visual representation (e.g., physically-based rendering vs. line sketching). We also have realistic auditory simulation. However, the "missing quadrant"—abstract auditory representation—remains largely unexplored.
When humans try to describe a sound, we often resort to vocal imitation. Interestingly, an effective imitation doesn't necessarily sound "exactly" like the target. For example, when imitating a motorboat, humans often ignore the loud, salient splashing noise (broadband noise) and instead mimic the low-pitched engine rumble because it is more distinctive. Existing automated methods for sound synthesis usually rely on direct feature matching, which fails to capture this human intuition for abstraction and communication.
Methods
The authors propose a three-stage model built from first principles:
- Vocal Tract Synthesis: They use a controllable "source-filter" model (implemented in the FAUST language) that simulates the physics of the human voice. They discretized the control space to create a library of ~161,000 possible human utterances.
- The Baseline (Feature Matching): Initially, the system tries to match a target sound (the "referent") by optimizing for 19 low-level auditory features (e.g., spectral centroid, flatness, loudness).
- The RSA Module (Communicative Reasoning): To move beyond literal matching, they apply the Rational Speech Acts (RSA) framework. This cognitive model treats communication as a recursive game: a "Level-2" speaker chooses an utterance by predicting how a "Level-1" listener would interpret it.
- Costs and Utilities: The model incorporates a "cost" for difficult vocalizations (e.g., very loud or rapid sounds) and a "utility" based on semantic similarity. If the listener guesses a "drill" instead of a "jackhammer," the speaker is partially satisfied because they are semantically related in the AudioSet ontology.
Datasets
- FSD50K: A library of over 51,000 audio clips used as the source for "referent" sounds.
- VocalImitationSet: A dataset of human-generated vocal imitations used to evaluate how well the model's outputs align with actual human behavior.
- AudioSet Ontology: A hierarchical classification of sounds used to calculate semantic "utility" (e.g., knowing that a "meow" and a "purr" both belong to the "Cat" category).
Key Insights and Ideas
- Informative > Salient: Humans don't imitate the loudest part of a sound; they imitate the part that best distinguishes that sound from others. The RSA framework mathematically captures this "pragmatic strengthening."
- Depiction as Optimization: Following the work of Frédo Durand, the authors view depiction (whether a sketch or a vocalization) as an optimization problem: find the stimulus that best induces the desired percept in the listener's mind.
- Zero-Shot Alignment: Remarkably, the model was never trained on human vocal data. Its high correlation with human behavior emerges purely from modeling the constraints of the vocal tract and the logic of human communication.
Key Results
- Human Alignment: The "full model" (including RSA and costs) achieved a correlation of $R^2 = 0.81$ with human phonetic features, significantly outperforming the feature-matching baseline ($R^2 \approx 0.57$).
- User Preference: In a 60-person user study, participants consistently preferred the imitations generated by the full model over the baseline. In some cases, users even preferred the model's synthetic imitations over actual human-generated ones.
- Retrieval Task: The system's "listener" model could identify the correct animal sound from a vocal imitation with 58% accuracy in a 3-way task (compared to 70% for humans and 46% for the baseline).
Limitations and Future Work
- Temporal Structure: The current method focuses on "sound textures" (stable sounds like rain or engines) and struggles with complex temporal structures like speech or melodic music.
- Phonetic Range: The vocal tract model lacks certain consonants (like the buzzing [z]), leading to divergences from human imitations of insects.
- Cultural Conventions: The model operates on "iconicity" (direct resemblance) and cannot currently predict culturally evolved onomatopoeia (like saying "ba-boom" for a heartbeat, which is acoustically a poor match but a strong cultural convention).
- Future Applications: The authors suggest this could lead to "sketch-based" interfaces for sound designers, allowing them to search sound libraries or synthesize effects using only their voice.
Summary updated