On Neural Phone Recognition of Mixed-Source ECoG Signals

Ahmed Hussen Abdelaziz, Shuo-Yiin Chang, Nelson Morgan, Erik Edwards, Dorothea Kolossa, Dan Ellis, David A. Moses, Edward F. ChangarXiv 2019 · arXiv

Introduction

The "cocktail party effect" describes the remarkable human ability to isolate and understand a single voice within a noisy, multi-speaker environment. While modern Automatic Speech Recognition (ASR) systems achieve high accuracy in quiet conditions, their performance often degrades significantly in the presence of competing speech. This research investigates the robustness of the human auditory cortex compared to algorithmic approaches by developing an improved Neural Speech Recognition (NSR) framework that decodes speech directly from electrocorticography (ECoG) signals.

Methodology: The NSR Framework

The study utilizes a hybrid Deep Neural Network/Hidden Markov Model (DNN/HMM) architecture to translate ECoG signals into phonetic sequences. ECoG data was acquired using a high-density electrode array placed over the posterior superior temporal gyrus (pSTG), a region critical for speech processing.

The proposed experimental framework for comparing Neural Speech Recognition (NSR) and Automatic Speech Recognition (ASR) performance.

To address the inherent challenges of neural decoding, two primary technical refinements were implemented:

  1. Manual Alignment Initialization: Rather than starting with a "flat" model, the researchers used a subset of manually transcribed data to initialize the ASR models, providing a more accurate baseline for training the NSR system.
  2. Optional Silence Paths: To account for the temporal "jitter" and biological lag between acoustic stimuli and neural processing, the training lattices included "optional silences" with a 0.9 skip probability. This allowed the model to handle gaps in neural activity without forcing incorrect phonetic alignments.

Experimental Setup and Datasets

The framework was evaluated using three primary data sources:

  • Coordinate Response Measure (CRM): A small-vocabulary corpus with a rigid grammar (e.g., "Ready [Call Sign] Go To [Color] [Number] Now").
  • Mixed-Source CRM: Created by overlaying male and female speaker recordings to simulate a two-speaker interference environment.
  • TIMIT: A standard acoustic-phonetic corpus used to augment training data and improve phonetic coverage.

Key Insights: Neural Robustness vs. Algorithmic Fragility

A central finding of this work is the disparity in how ASR and NSR systems handle noise. While ASR systems rely on clear acoustic features, the NSR system leverages the brain's inherent filtering mechanisms.

Comparison of target (a) and mixed (b) acoustic spectrograms against the corresponding neural feature index (c), showing brain activity during speech processing.

Visualization of spatiotemporal neural representations revealed intensive activity specifically when the subject attended to target keywords (call signs, colors, and numbers). This suggests that the auditory cortex actively suppresses background interference, a process the NSR system attempts to capture.

Results and Discussion

The performance gap between single-speaker and mixed-speaker scenarios highlights the robustness of neural signals:

  • ASR Performance: In quiet conditions, ASR achieved a Phone Error Rate (PER) of 4.01%. However, in the mixed-speaker scenario, the PER surged to 54.27%, a relative increase of 93%.
  • NSR Performance: Although the NSR system had a higher baseline PER (56.75%), it demonstrated superior stability. In the mixed-speaker environment, the PER only increased to 64.54%, representing a modest 12% relative degradation.

Furthermore, the inclusion of GMM/HMM-based alignments and optimized silence paths resulted in a 4.5% relative reduction in error compared to previous NSR baselines.

Conclusion and Future Work

While the current NSR system's absolute accuracy is limited by the availability of high-density ECoG data, its relative robustness to noise provides a blueprint for future noise-resilient ASR. Future research will explore how different acoustic environments, such as reverberation, impact neural processing. The ultimate goal is to reverse-engineer biological filtering mechanisms to create artificial systems that match the human brain's ability to decode speech in complex auditory landscapes.

Summary updated

BibTeX
@misc{abdelaziz2019neural,
  author = {Ahmed Hussen Abdelaziz and Shuo-Yiin Chang and Nelson Morgan and Erik Edwards and Dorothea Kolossa and Dan Ellis and David A. Moses and Edward F. Chang},
  title = {{On Neural Phone Recognition of Mixed-Source ECoG Signals}},
  year = {2019},
  eprint = {1912.05869},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.05869},
}
Figure