AudioViewer: Learning to Visualize Sound

Yuchi Zhang, Willis Peng, Bastian Wandt, Helge RhodinarXiv 2020 · OpenReview

TL;DR

This paper introduces AudioViewer, a novel system that translates audio into dynamic video visualizations in real-time, aiming to help individuals with hearing impairments perceive sound visually. Unlike previous methods that focus on specific tasks like lip-reading or text translation, AudioViewer learns to map general environmental sounds and speech to natural images (like faces or numbers) using unpaired data and high-level constraints, demonstrating improved distinguishability and learnability compared to existing approaches.

What Problem Does This Paper Address?

The core challenge this paper tackles is sensory substitution for individuals with hearing impairments. Specifically, it addresses the long-standing goal of enabling deaf and hard-of-hearing (DHH) individuals to "see" sound by translating audio content into visual representations. Existing methods often fall short: lip-syncing only captures a fraction of speech, text translation loses vocal nuances, and traditional visual tools like spectrograms produce unnatural and difficult-to-interpret images. The authors aim to create a more intuitive, low-level audio-to-video translation that works for both speech and environmental sounds, without requiring paired audio-visual data for training.

Methods

The authors propose AudioViewer, a system built upon unpaired image-to-image translation techniques, enhanced with novel constraints to learn a mapping from audio to video. Key methodological contributions include:

  • Unsupervised Learning with Cycle Consistency: To learn the audio-to-video mapping without paired examples, they leverage cycle consistency, ensuring that translating from audio to video and back to audio (or from video to audio and back to video) reconstructs the original signal.
  • High-Level Constraints for Naturalness: To make the visualizations interpretable and natural, they enforce constraints such as temporal smoothness (preventing jerky visual changes) and a focus on patterns humans naturally recognize (like faces and digits).
  • Content-Style Disentanglement for Speech: For speech, they go a step further by disentangling the "content" (what is being said) from the "style" (gender, dialect). This allows for more nuanced visualizations and better understanding of vocal characteristics.
  • Audio Encoding: They use a SpeechVAE model to encode audio into a latent representation, and a visual decoder (like DFC-VAE or Soft-IntroVAE) to generate images from this latent code.
  • Translation Network (T): To bridge potentially mismatched latent spaces between audio and video models, they introduce a translation network that maps audio latent codes to visual latent codes.

Datasets

  • TIMIT dataset: Used for training the audio encoder, specifically for learning speech embeddings. It contains phonetic and speaker annotations, crucial for the content-style disentanglement.
  • CelebA-HQ, CelebA, and MNIST datasets: Used for training the visual decoders to generate realistic faces and digits.
  • ESC-50 dataset: Used to evaluate the generalization capability of the trained model on environmental sounds, demonstrating its applicability beyond speech.

Key Insights and Ideas

  • Unpaired Learning is Key: The absence of readily available paired audio-video data for general sounds necessitates unsupervised learning techniques, and cycle consistency proves effective.
  • Faces and Digits are Good Visualizers: The human studies suggest that faces and digits are well-suited for visualizing high-dimensional audio features, allowing humans to parse and distinguish sounds more effectively than abstract representations.
  • Content-Style Separation Enhances Understanding: Disentangling speech content from style (gender, dialect) leads to more informative visualizations and improves the ability to distinguish speakers.
  • Smoothness Matters for Perception: Enforcing temporal smoothness in the generated video significantly improves the perceptual quality and ease of understanding.
  • Low-Level Mapping is Crucial for Learning: Unlike high-level translations (like text), a low-level audio-to-visual mapping is more appropriate for learning speech production and understanding nuances, especially for children.

Key Results

  • Distinguishability: Human studies showed that AudioViewer visualizations allowed users to distinguish between different sounds with high accuracy (e.g., 87.9% for matching tasks with high-res CelebA content model). This significantly outperformed mel spectrograms (66.2%) and lip-sync methods (47.7%).
  • Learnability: In a learning study, participants using AudioViewer achieved a final accuracy of 87.0% after 16 rounds, compared to 57.8% for the mel spectrogram baseline, indicating better learnability of the visual representations.
  • Content vs. Style: Disentangling content and style improved accuracy for distinguishing speakers of different sexes (from 43.3% to 78.0%) and dialects (from 39.6% to 56.7%).
  • Environmental Sounds: The model demonstrated the ability to visualize environmental sounds (like bird chirps and alarm clocks) even though it was primarily trained on speech.
  • Runtime: The system is capable of real-time inference (5 ms/frame on a high-end GPU).

Limitations and Future Work

  • Ambiguity with Similar Sounds: The paper acknowledges that distinguishing very similar sounds (like "that" and "there") can still be challenging, even with learning.
  • Environmental Sound Generalization: While promising, the SNR for environmental sounds was lower than for speech, suggesting room for improvement in generalizing to diverse soundscapes.
  • Future Directions:
    • Improved Expressiveness: Making the visual language more intuitive and expressive.
  • Hybrid Approach: Mapping speech to lip motion for better naturalness while using the general encoding for other sounds.
  • Longer Learning Sessions: Investigating if longer learning periods can overcome remaining ambiguities.
  • Domain Expert Control: Developing psychophysical studies with domain experts to mitigate risks and ensure long-term participant well-being.

Summary updated

Abstract

A long-standing goal in the field of sensory substitution is to enable sound perception for deaf and hard of hearing (DHH) people by visualizing audio content. Different from existing models that translate to hand sign language, between speech and text, or text and images, we target immediate and low-level audio to video translation that applies to generic environment sounds as well as human speech. Since such a substitution is artificial, without labels for supervised learning, our core contribution is to build a mapping from audio to video that learns from unpaired examples via high-level constraints. For speech, we additionally disentangle content from style, such as gender and dialect. Qualitative and quantitative results, including a human study, demonstrate that our unpaired translation approach maintains important audio features in the generated video and that videos of faces and numbers are well suited for visualizing high-dimensional audio features that can be parsed by humans to match and distinguish between sounds and words. Code and models are available at https://chunjinsong.github.io/audioviewer
BibTeX
@misc{zhang2020audioviewer,
  author = {Yuchi Zhang and Willis Peng and Bastian Wandt and Helge Rhodin},
  title = {{AudioViewer: Learning to Visualize Sound}},
  year = {2020},
  eprint = {2012.13341},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.13341},
}
Figure