Realtime Generation of Audible Textures Inspired by a Video Stream

Simone Mellace, Jérôme Guzzi, Alessandro Giusti, Luca Maria GambardellaAAAI 2019 · DOI

Introduction

This research presents a system for generating real-time "audible textures"—immersive, continuous soundscapes—driven by a live video feed. Unlike traditional image-to-sound generation that focuses on literal realism (e.g., the sound of a specific impact), this approach aims to evoke the "feeling" or atmosphere of a visual scene. By mapping high-level semantic features from a camera to a granular audio synthesizer, the system creates ambient environments like the hum of an office or the rustle of a forest that respond dynamically to visual input.

System Architecture

The system bridges modern deep learning with digital signal processing through a four-stage pipeline:

  1. Semantic Feature Extraction: A ResNet50 CNN, pre-trained on ImageNet, extracts a 2048-dimensional vector from its penultimate layer. This vector captures high-level semantic concepts, allowing the system to recognize conceptual similarities (e.g., a "goose" being similar to a "duck").
  2. Reference Dataset: The system utilizes a pre-built dataset of aligned video and audio. CNN feature vectors are pre-computed for frames within these reference videos.
  3. The Mapping Loop: In real-time, the system calculates the Euclidean distance between the current camera frame's feature vector and the reference vectors. This distance determines a probability distribution $p(t)$, where semantically similar reference frames are assigned higher probabilities.
  4. Granular Synthesis: The audio engine samples short "granules" (50–100 ms) from the reference audio based on the calculated probability distribution. By overlaying multiple granules at a high frequency (100 Hz), it creates a smooth, non-looping texture that retains the "character" of the source material without being a literal loop.
Diagram of the semantic mapping process between live video frames and reference audio timestamps.

Implementation and Data

The project was implemented using Python, Keras for deep learning, and the Pyo library for digital signal processing. The reference dataset consists of hand-selected YouTube videos (e.g., nature documentaries) where the audio track is meaningfully related to the visual content, avoiding voice-overs or unrelated music to maintain semantic integrity.

Results and Insights

The system achieves real-time performance, providing immediate audio feedback on mobile devices. A key insight is the use of granular synthesis as a form of "audio blur," which allows for seamless transitions between textures. As a user pans the camera, the sound shifts through related semantic categories (e.g., different types of greenery) rather than jumping abruptly.

Real-time demonstration of the system generating audible textures based on a mobile camera feed.

Qualitatively, the system offers an intuitive look into neural network perception. Users can "hear" how the CNN perceives similarity, such as the difference in sonic texture between a selfie and a landscape.

Limitations

The system's effectiveness is currently bound by the diversity of its reference dataset. If the camera encounters a scene with no semantic match in the reference data, the audio may feel disconnected. Future work could involve increasing sampling density for more nuanced responses and automating the selection of "clean" reference videos to reduce manual curation.

Summary updated

BibTeX
@inproceedings{mellace2019realtime,
  author = {Simone Mellace and Jérôme Guzzi and Alessandro Giusti and Luca Maria Gambardella},
  title = {{Realtime Generation of Audible Textures Inspired by a Video Stream}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.33019865},
}
Figure