Intelligible speech synthesis from neural decoding of spoken sentences

Gopala K. Anumanchipalli, Josh Chartier, Edward F. ChangbioRxiv 2018 · DOI

TL;DR

Researchers have developed a novel neural decoder that can translate brain activity directly into intelligible synthesized speech, even from silently mimed speech. This breakthrough offers a promising new avenue for restoring communication for individuals with severe neurological impairments, achieving natural speaking rates and capturing prosodic elements.

What Problem Does This Paper Address?

For individuals who have lost the ability to speak due to neurological conditions like paralysis, current communication devices often rely on slow, laborious methods like spelling out words letter by letter. While these systems improve quality of life, they fall far short of natural speech rates (around 10 words/minute compared to 150 words/minute). The challenge is to create a system that can decode brain signals related to speech production and synthesize fluent, intelligible speech at a natural pace, effectively bypassing the damaged motor pathways. This paper tackles this by developing a neural decoder that leverages the complex, millisecond-level control signals involved in vocal tract articulation.

Methods

The core of this work is a two-stage recurrent neural network (RNN) decoder.

  • Stage 1: Articulatory Kinematics Decoding: A bidirectional long short-term memory (bLSTM) network takes high-density electrocorticography (ECoG) signals from the brain and decodes intermediate articulatory kinematic features. These features represent the physical movements of the vocal tract (e.g., tongue, lips, jaw) that produce speech sounds. This stage explicitly models the physiological process of speech generation.
  • Stage 2: Acoustic Synthesis: A second bLSTM network takes the decoded articulatory features and transforms them into acoustic features. These acoustic features are then used to synthesize the final audible speech waveform.

This "biomimetic" approach, which explicitly models articulatory movements, is a key innovation. It aims to simplify the complex mapping from neural activity to sound by first capturing the underlying physical movements.

Datasets

The study used data from three human participants undergoing intracranial monitoring for epilepsy treatment. They provided:

  • High-density ECoG recordings.
  • Audible speech recordings.
  • Silently mimed speech recordings.

Specific datasets mentioned for training include:

  • MOCHA-TIMIT database (for reading aloud sentences).
  • Passages from various stories.
  • Picture scenes descriptions.
  • Free response interviews.

The paper also mentions using an existing annotated speech database (Wall Street Journal Corpus) for training speaker-independent models to infer articulatory kinematics.

Key Insights and Ideas

  • Articulatory Kinematics as a Crucial Intermediate: Explicitly modeling vocal tract movements as an intermediate step significantly improves speech decoding performance and reduces the amount of training data needed, compared to directly decoding acoustics from ECoG. This suggests that the brain's representation of articulation is a more direct and simpler target for decoding than the final acoustic output.
  • Decoding Silent Speech: The decoder can generate intelligible speech even from brain activity recorded while participants silently mimed speech. This is a critical step towards communication devices that can function even when vocalization is impossible.
  • Limited Data, Robust Performance: The system achieves robust decoding performance with as little as 25 minutes of training data, which is a significant advantage for patients who may have limited ability to provide extensive training data.
  • Natural Speech Elements: The synthesized speech captures not only the words but also prosodic elements like pitch and speech envelope, contributing to more natural-sounding communication.

Key Results

  • Intelligible Speech Synthesis: Naïve listeners could accurately identify decoded sentences, with a median accuracy of 83% in a 10-choice identification task.
  • Spectral Distortion: The Mel-Cepstral Distortion (MCD) for decoded speech was significantly lower than chance-level decoding and comparable to state-of-the-art methods using facial EMG. Median MCD scores were around 5.14-5.55 dB.
  • Training Data Efficiency: Performance improved with more training data, but robust results were achieved with just 25 minutes. Direct ECoG-to-acoustic decoding without the articulatory intermediate step showed a substantial performance drop (0.54 dB MCD increase) when using the full dataset.
  • Silent Speech Performance: Decoding of silently mimed speech was inferior to audible speech (30% MCD difference) but still significantly better than chance, demonstrating the possibility of decoding speech features even without vocalization.

Limitations and Future Work

  • Data Scarcity: While the system is efficient with limited data, obtaining even 25 minutes of high-quality ECoG data can be challenging for patients with severe paralysis.
  • Absence of Voicing/Laryngeal Activation: The performance gap in silent speech decoding is partly attributed to the absence of voicing and laryngeal activation signals, which are crucial for audible speech.
  • Future Directions: The authors suggest that future improvements could come from modeling higher-order linguistic and planning goals, which could further enhance the naturalness and complexity of the synthesized speech. They also highlight the potential for this articulatory-based approach to serve as a new foundation for speech decoding.

Summary updated

Abstract

Abstract The ability to read out, or decode, mental content from brain activity has significant practical and scientific implications 1 . For example, technology that translates cortical activity into speech would be transformative for people unable to communicate as a result of neurological impairment 2,3,4 . Decoding speech from neural activity is challenging because speaking requires extremely precise and dynamic control of multiple vocal tract articulators on the order of milliseconds. Here, we designed a neural decoder that explicitly leverages the continuous kinematic and sound representations encoded in cortical activity 5,6 to generate fluent and intelligible speech. A recurrent neural network first decoded vocal tract physiological signals from direct cortical recordings, and then transformed them to acoustic speech output. Robust decoding performance was achieved with as little as 25 minutes of training data. Naïve listeners were able to accurately identify these decoded sentences. Additionally, speech decoding was not only effective for audibly produced speech, but also when participants silently mimed speech. These results advance the development of speech neuroprosthetic technology to restore spoken communication in patients with disabling neurological disorders.
BibTeX
@misc{anumanchipalli2018intelligible,
  author = {Gopala K. Anumanchipalli and Josh Chartier and Edward F. Chang},
  title = {{Intelligible speech synthesis from neural decoding of spoken sentences}},
  year = {2018},
  doi = {10.1101/481267},
  howpublished = {bioRxiv},
}

Loading related content...

Figure