Onset-based Sound Segmentation

Leslie S. SmithNeurIPS 1995

Introduction

Traditional speech recognition and sound analysis often rely on Fourier transforms, which analyze sound in fixed time windows (usually 25–75 ms). While effective for stationary signals, this approach struggles with the fluid, continuous nature of speech and is notoriously sensitive to background noise.

In contrast, the human auditory system does not rely on fixed windows. It possesses incredible temporal resolution and can easily isolate a specific voice in a noisy room. This paper introduces a biologically inspired technique for data-driven sound segmentation, mimicking how the mammalian ear and brainstem detect the "start" (onset) and "end" (offset) of sounds. By using a network of integrate-and-fire neurons to cluster signals across different frequencies, the system achieves robust sound segmentation that remains effective even in the presence of heavy background noise.

Methods: A Biologically Inspired Pipeline

The proposed system follows a multi-stage pipeline modeled after the early auditory pathway:

  1. Cochlear Filtering: The audio is passed through a bank of bandpass filters that mimic the frequency response of the cochlea.
  2. Hair Cell Simulation: The signals are rectified to model how inner hair cells convert mechanical vibrations into electrical impulses.
  3. The Onset-Offset Filter: This is a crucial innovation. The system uses a "Difference of Gaussians" (DoG) filter—a technique commonly used in computer vision for edge detection. In this context, it treats the sudden rise or fall of sound energy as a "temporal edge."
  4. Logarithmic Compression: To handle the massive dynamic range of sound, the signals are compressed, mimicking biological sensitivity.
  5. Integrate-and-Fire Neural Network: The final stage uses a network of artificial neurons. These neurons accumulate input over time and "fire" a spike when they reach a threshold. Crucially, these neurons are interconnected; if one neuron fires because it detects an onset in its frequency band, it encourages its neighbors in adjacent bands to fire as well. This creates a synchronized "burst" of spikes that marks a clear segmentation boundary.

Key Insights

  • Sound as an Image: One of the most compelling ideas is applying Marr and Hildreth’s vision theories to audio. By treating an onset as a "white-to-black edge" in time, the author successfully adapts spatial edge detection to the temporal domain.
  • Asymmetry of Onsets and Offsets: The paper notes that onsets are usually sharp and brief (due to the physics of sound production), while offsets are gradual (due to echoes and vibration decay). The system accounts for this by using different "refractory periods" (cool-down times) for onset and offset neurons.
  • Neural Synchronization: The use of lateral connections in the neural network allows the system to "clean up" messy data. Instead of having dozens of tiny, uncoordinated detections across different frequencies, the network forces them to cluster into a single, unambiguous segmentation point.

Datasets and Key Results

The system was evaluated on specific, challenging recordings to test its limits:

  • Speech: A recording of the phrase "Neural information processing systems" spoken at different speeds.
  • Noisy Speech: The same phrase layered with recorded motorbike noise.
  • Music: A recording of a "slurred" flute sound, where notes transition smoothly into one another without silence.
Waveforms and neural spike segmentations for clean and noisy speech.

Speech and Noise Robustness: The system successfully broke down the clean test utterance into 19 segments at normal speed and 25 segments when spoken slowly. While not perfectly aligned with linguistic phonemes, the segments provided a consistent, data-driven representation of the speech. Furthermore, the system remained highly effective at finding speech segments even when buried under motorbike noise—a scenario where traditional Fourier-based methods often fail.

Segmentation of slurred flute notes using the integrate-and-fire network.

Musical Precision: In previous work, segmenting slurred flute notes proved difficult because the acoustic changes were too smooth. The addition of the integrate-and-fire network allowed the system to detect these subtle transitions accurately, marking distinct notes even without intervening silence.

Limitations and Future Work

The author acknowledges that while the system reliably finds where the segments are, it does not yet fully utilize the frequency information contained within those segments.

Future directions include:

  • Streaming: Combining this segmentation with "streaming," where the system groups specific frequency bands that share the same amplitude modulation. This would allow the system to separate two different sound sources (like two people talking) simultaneously.
  • Silicon Implementation: Because the model uses simple integrate-and-fire neurons and standard filters, it is highly suitable for implementation in specialized neuromorphic hardware (silicon chips) for real-time, low-power processing.

Summary updated

BibTeX
@inproceedings{smith1995onset,
  author = {Leslie S. Smith},
  title = {{Onset-based Sound Segmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1995},
}
Figure