AudioViewer: Learning to Visualize Sounds

Chunjin Song, Yuchi Zhang, Willis Peng, Parmis Mohaghegh, Bastian Wandt, Helge RhodinWACV 2023 · OpenReview

Abstract

A long-standing goal in the field of sensory substitution is enabling sound perception for deaf and hard of hearing (DHH) people by visualizing audio content. Different from existing models that translate to hand sign language, between speech and text, or text and images, we target immediate and low-level audio to video translation that applies to generic environment sounds as well as human speech. Since such a substitution is artificial, without labels for supervised learning, our core contribution is to build a mapping from audio to video that learns from unpaired examples via high-level constraints. For speech, we additionally disentangle content from style, such as gender and dialect. Qualitative and quantitative results, including a human study, demonstrate that our unpaired translation approach maintains important audio features in the generated video and that videos of faces and numbers are well suited for visualizing high-dimensional audio features that can be parsed by humans to match and distinguish between sounds and words.

Introduction

For the deaf and hard-of-hearing (DHH) community, current sensory substitution tools often fall into two extremes: high-level abstractions like speech-to-text or low-level raw data like spectrograms. While text-to-speech is intuitive, it often strips away vocal nuances—such as gender, dialect, and emotion—and fails entirely for environmental sounds like doorbells or barking dogs. Conversely, raw spectrograms are comprehensive but incredibly difficult for humans to interpret in real-time.

AudioViewer seeks a middle ground: a low-level mapping that preserves the "texture" of sound while using visual patterns—specifically human faces—that our brains are evolutionarily hard-wired to process efficiently. By translating audio into real-time streams of natural images, it provides a way to "see" the style and tone of sound.

The AudioViewer pipeline: translating audio signals into visual streams of faces or digits via a latent space trajectory.

Methodology: Unpaired Translation

The core challenge in audio-to-video visualization is the lack of "ground truth" datasets pairing specific sounds with specific visual distortions. AudioViewer addresses this through an unpaired translation approach using the following components:

  1. Cross-Modal VAEs: Separate Variational Autoencoders are trained for audio (Mel-spectrograms) and video (faces or digits).
  2. Latent Space Mapping: A translation network ($T$) maps the audio latent space to the visual latent space.
  3. Self-Supervised Constraints: To train without paired data, the model enforces Cycle Consistency (ensuring back-translation preserves the original sound), Temporal Smoothness (to prevent flickering), and Distance Preservation (ensuring acoustically similar sounds remain visually similar).
Unpaired translation architecture showing the mapping between audio and visual latent spaces with self-supervised constraints.

For speech, the model further disentangles content (phonemes) from style (speaker identity), allowing users to focus on words or tone independently.

Datasets and Evaluation

The researchers evaluated the model across several diverse datasets to ensure robustness:

  • TIMIT: Used for learning acoustic-phonetic speech embeddings.
  • CelebA & CelebA-HQ: Large-scale face datasets used to generate the visual output.
  • MNIST: Hand-written digits, providing a simpler alternative to faces for visualization.
  • ESC-50: Environmental recordings used to test generalization beyond human speech.

Key Results and Insights

AudioViewer demonstrates that the human face can serve as a highly effective high-dimensional data display. Because humans have specialized brain regions for recognizing minute facial changes, these visualizations can be "read" much faster than complex graphs.

  • High Distinguishability: In a human study, participants matched visualizations to the correct sounds with 86.4% accuracy using the high-resolution face model.
  • Rapid Learnability: Users' accuracy improved from near-random to 87% after just 16 attempts, suggesting that humans can quickly adapt to this new visual language.
  • Real-Time Performance: The system is highly efficient, with inference times as low as 5ms to 7.6ms per frame on a standard consumer GPU (RTX 2080 Ti).
  • Superiority to Lip-Syncing: While traditional lip-syncing often makes different phonemes (like 'd' and 't') look identical, AudioViewer's visualizations remain distinct.

Limitations and Future Work

While promising, AudioViewer requires an initial learning curve as users must learn to associate specific facial distortions with sounds. Some phonetic ambiguities remain, and environmental sound visualization currently loses some high-frequency detail due to the speech-centric training of the audio models. Future work aims to bridge the gap between these artificial mappings and natural lip motion, potentially creating a hybrid system that combines intuitive lip-reading with high-detail data encoding.

Summary updated

Abstract

A long-standing goal in the field of sensory substitution is enabling sound perception for deaf and hard of hearing (DHH) people by visualizing audio content. Different from existing models that translate to hand sign language, between speech and text, or text and images, we target immediate and low-level audio to video translation that applies to generic environment sounds as well as human speech. Since such a substitution is artificial, with-out labels for supervised learning, our core contribution is to build a mapping from audio to video that learns from unpaired examples via high-level constraints. For speech, we additionally disentangle content from style, such as gender and dialect. Qualitative and quantitative results, including a human study, demonstrate that our unpaired translation approach maintains important audio features in the generated video and that videos of faces and numbers are well suited for visualizing high-dimensional audio features that can be parsed by humans to match and distinguish between sounds and words. Project website: https://chunjinsong.github.io/audioviewer
BibTeX
@inproceedings{song2023audioviewer,
  author = {Chunjin Song and Yuchi Zhang and Willis Peng and Parmis Mohaghegh and Bastian Wandt and Helge Rhodin},
  title = {{AudioViewer: Learning to Visualize Sounds}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=UCB75vZlPn},
}
Figure