DeepWave: A Recurrent Neural-Network for Real-Time Acoustic Imaging

Matthieu Simeoni, Sepand Kashani, Paul Hurley, Martin VetterliNeurIPS 2019

Acoustic cameras allow us to "see" sound by overlaying a heatmap of audio intensity onto a live video feed. This technology is invaluable for everything from automotive engineering to noise pollution monitoring. However, traditional acoustic imaging faces a strict tradeoff between speed and image clarity.

Standard devices rely on Delay-And-Sum (DAS) beamforming. While DAS is incredibly fast, it produces highly blurred images due to the physical limitations of microphone arrays. To achieve sharper images, researchers typically use Compressed Sensing (CS)—mathematical optimization techniques that "deblur" the sound field. Unfortunately, these iterative algorithms are computationally heavy, often taking hundreds of milliseconds per frame. This makes them entirely unsuitable for real-time video, which requires updates 12 to 25 times per second.

Furthermore, standard deep learning tools like Convolutional Neural Networks (CNNs) struggle to solve this problem because acoustic data is complex-valued and mapped onto a spherical geometry, rather than a flat 2D grid of pixels.

To bridge the gap between fast-but-blurry DAS and sharp-but-slow optimization, researchers have introduced DeepWave, a novel recurrent neural network (RNN) architecture that achieves high-resolution acoustic imaging in real-time.

The DeepWave Architecture

DeepWave achieves its speed and accuracy by "unrolling" a physics-based optimization algorithm into a neural network.

The DeepWave architecture, illustrating the unrolled optimization layers and graph-based processing.

Key technical innovations of the DeepWave model include:

  • Algorithm Unrolling: DeepWave is built on the "Learned Iterative Soft-Thresholding Algorithm" (LISTA) framework. It takes a standard iterative solver (Proximal Gradient Descent) and converts each iteration into a discrete layer of a neural network. This allows the network to learn "shortcuts" to a high-quality solution in just 5 to 6 layers, whereas the original mathematical solver might require over 50 iterations.
  • Graph Signal Processing (GSP): Because acoustic maps are spherical, standard 2D convolutions fail. DeepWave utilizes graph Laplacians to implement filtering operations directly on the irregular geometry of a spherical mesh.
  • Physically-Inspired Parametrization: Instead of relying on millions of random, opaque weights, the network’s internal structures (such as deblurring and back-projection operators) are strictly constrained by the physics of sound propagation. This makes the model "parsimonious"—it requires very few parameters and is highly efficient to train.
  • Smart Initialization: Rather than starting with randomized weights, the network is initialized using "oracle" parameters derived from underlying acoustic physics. This prevents the "exploding gradient" problem common in RNNs and ensures rapid convergence.

Datasets and Evaluation

The research team evaluated DeepWave across three distinct datasets to ensure robustness:

  1. Real-World: Recordings of 8 people speaking in a conference room, captured by a 48-element "Pyramic" microphone array.
  2. Real-World/Augmented: 2,700 template recordings from an anechoic chamber, combined to simulate complex, multi-source acoustic environments.
  3. Simulated: A synthetic dataset of narrow-band point sources designed to test the network’s ability to generalize to unseen spatial directions.

Key Results: Speed Meets Resolution

DeepWave successfully shatters the traditional speed-resolution tradeoff, proving that physics-informed neural networks can deliver the best of both worlds.

Comparison of DAS and DeepWave sound fields, demonstrating DeepWave's superior resolution and contrast.
  • Resolution: DeepWave improved angular resolution by approximately 27% compared to standard DAS beamforming. Visual inspections of the sound fields show significantly darker backgrounds and sharper, more localized "blobs" for sound sources, making it vastly easier to distinguish between multiple active speakers in a room.
  • Speed: DeepWave processes a single frame in just 6.5 ms. While DAS is slightly faster (2.0 ms), both easily clear the threshold required for real-time video. In stark contrast, the high-resolution optimization baseline (APGD) took 211 ms per frame—far too slow for live applications.

Limitations and Future Work

While DeepWave represents a massive leap forward for acoustic cameras, the authors note a few areas for future refinement:

  • Frequency Dependence: The current model is trained on specific frequency bands. Developing a "frequency-invariant" architecture would allow a single model to seamlessly handle all types of sound.
  • Temporal Consistency: Currently, DeepWave processes each frame in isolation. Making the network "time-aware"—by feeding the output of one frame as the initial guess for the next—could drastically improve visual stability in live video feeds.
  • Array Specificity: The network's weights are tied to the specific geometry of the microphone array used during training (e.g., the Pyramic array). Adapting a trained model to a completely different hardware shape currently requires retraining or fine-tuning.

By treating physics as a structural prior rather than an afterthought, DeepWave proves that deep learning can be both highly interpretable and blisteringly fast, paving the way for the next generation of real-time acoustic imaging.

Summary updated

BibTeX
@inproceedings{simeoni2019deepwave,
  author = {Matthieu Simeoni and Sepand Kashani and Paul Hurley and Martin Vetterli},
  title = {{DeepWave: A Recurrent Neural-Network for Real-Time Acoustic Imaging}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure