Pattern Playback in the 90s

Malcolm SlaneyNeurIPS 1994

In auditory science, researchers rely on sophisticated mathematical models to represent how the human ear and brain process sound. While simple tools like spectrograms are ubiquitous, tackling complex challenges—such as separating a single voice from a noisy cocktail party—requires richer, neurophysiologically inspired representations like the correlogram.

However, these advanced models introduce a significant hurdle: they are highly non-linear and "lossy." The process involves steps like half-wave rectification (discarding the negative portion of a sound wave) and automatic gain control (AGC, which compresses volume). Historically, converting a sound into these representations was a one-way trip. Researchers could analyze the visual data, but they couldn't easily invert it back into a listenable audio waveform. Without this inversion capability, qualitatively evaluating whether a noise-reduction algorithm actually sounds natural—or if it introduces robotic artifacts—is nearly impossible.

Overview of the auditory representation pipeline, showing the forward and inverse paths between the waveform, cochleagram, and correlogram.

This paper introduces a robust method for "inverting" these complex auditory representations back into audible sound, providing a vital tool for testing sound separation and speech enhancement algorithms.

Reversing the Auditory Pipeline

The authors break the inversion process down into two primary stages: Cochleagram Inversion and Correlogram Inversion.

1. Cochleagram Inversion

The first major challenge is reversing a three-stage cochlear model that consists of a filterbank, a detector (half-wave rectifier), and AGC.

  • Inverting AGC: Because the AGC is controlled by its own output, the authors track the output history to mathematically "divide out" the applied gain.
  • Recovering the "Lost" Wave: To recover the negative portions of the signal lost during half-wave rectification, the authors employ Projections onto Convex Sets (POCS). This technique leverages the mathematical property that, because each channel is frequency-limited, there is only one "smooth" way to fill in the missing gaps.
  • Filterbank Inversion: Finally, to combine the separate frequency channels back into a single cohesive wave, the signal is run through the filters backwards. This clever trick effectively cancels out the phase delays introduced during the initial forward pass.
Block diagram of the iterative cochleagram inversion process, highlighting the loop between temporal and spectral projections to recover lost signal data.

2. Correlogram Inversion

The correlogram adds another layer of complexity by calculating the periodicity (autocorrelation) of each cochlear channel.

To invert this, the authors note that the Fourier transform of an autocorrelation yields a power spectrum. By applying this row by row, they convert the correlogram into a series of spectrograms. The missing phase information is then estimated using an iterative algorithm similar to Griffin-Lim. A key efficiency hack introduced here is using the recovered phase from one cochlear channel to initialize the next. Because adjacent channels overlap significantly, this "warm start" makes the process much faster and more consistent.

Datasets and Key Results

The researchers evaluated their inversion algorithms using both synthetic signals (a 200Hz impulse train to test mathematical precision) and real-world human speech (the syllable "tap" from the TIMIT Speech Database).

The results demonstrated near-perfect reconstruction. For both the impulses and the speech samples, the model produced waveforms that were perceptually identical to the originals, successfully maintaining the integrity of glottal pulses and sharp consonant onsets. Furthermore, the iterative inversion process proved highly efficient. Thanks to the optimized spectral projection, high-quality results were sometimes achieved with zero iterations, and nearly perfect results required just ten.

Insights and Future Directions

This work proves that despite being a non-linear, neurophysiologically plausible representation, the correlogram preserves enough salient information to perfectly reconstruct the original acoustic signal. It also highlights that phase is the hardest puzzle in auditory inversion; using cross-correlation to align new segments during reconstruction significantly improved output quality. Interestingly, the authors found that if the AGC is not inverted, the resulting sound naturally emphasizes onsets and de-emphasizes steady vowels, closely mimicking human auditory perception.

While highly successful, the model has limitations. It cannot recover information near 0 Hz (the "DC gap"), as this is naturally lost in the auditory system. Additionally, AGC inversion becomes increasingly difficult with louder inputs and more aggressive compression.

Looking forward, the authors identify the reconstruction of sound from partial correlograms as the next major frontier. Successfully inverting a signal after a noise-cancellation algorithm has "zeroed out" specific parts of it remains the holy grail for practical speech enhancement.

Summary updated

BibTeX
@inproceedings{slaney1994pattern,
  author = {Malcolm Slaney},
  title = {{Pattern Playback in the 90s}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1994},
}
Figure