Incorporating Symbolic Sequential Modeling for Speech Enhancement

Chien-Feng Liao, Yu Tsao, Xugang Lu, Hisashi Kawai

TL;DR

This paper introduces a novel speech enhancement (SE) system that incorporates "linguistic constraints" by learning abstract symbolic representations of speech, inspired by phoneme-like structures. This approach, using a Vector Quantized Variational Autoencoder (VQ-VAE) and a U-Net architecture with multi-head attention, significantly improves speech quality and intelligibility compared to traditional SE methods.

What Problem Does This Paper Address?

Speech enhancement (SE) aims to restore noisy speech to a cleaner version. While deep learning models have shown great promise, they primarily focus on learning the acoustic mapping between noisy and clean speech. However, human listeners naturally leverage their understanding of language (a "language model") to better interpret and reconstruct degraded speech. This paper argues that current SE systems miss out on this crucial linguistic information. The challenge is to incorporate this linguistic knowledge into SE models without requiring explicit transcriptions during training, which is often impractical.

Methods

The core innovation is the addition of a symbolic encoder to a standard U-Net based SE architecture. This symbolic encoder uses a Vector Quantized Variational Autoencoder (VQ-VAE) to learn discrete, symbolic representations from acoustic features (specifically MFCCs). These symbolic sequences are designed to capture high-level, phoneme-like content.

Here's a breakdown:

  • VQ-VAE for Symbolic Representation: The VQ-VAE learns a codebook of "prototype vectors." Input acoustic features are encoded into hidden vectors, which are then replaced by the closest prototype vector from the codebook. This process results in a sequence of discrete symbols. A "commitment loss" is used to encourage the encoder to produce vectors close to the prototypes.
  • U-Net Architecture: A standard U-Net is used for the primary acoustic mapping from noisy speech to clean speech.
  • Multi-Head Attention: This mechanism is employed to effectively integrate the symbolic sequence information into the U-Net decoder. It allows the decoder to attend to relevant parts of the symbolic sequence at different stages of reconstruction.
  • Joint Training: The U-Net and the symbolic encoder are trained end-to-end, allowing them to learn collaboratively without needing pre-defined transcriptions. The primary loss is the mean-squared error (MSE) between the enhanced and clean speech, with an additional commitment loss for the VQ-VAE.

Datasets

  • TIMIT Database: Used for training and evaluation. The authors created a large multi-condition training set by corrupting TIMIT utterances with 100 noise types at various Signal-to-Noise Ratios (SNRs). A validation set was also created with unseen noise types and SNRs. The core test set of TIMIT was used, along with additional unseen noise types from the NOISEX-92 corpus.

Key Insights and Ideas

  • Linguistic Knowledge Enhances SE: The paper strongly supports the intuition that understanding the linguistic structure of speech is beneficial for SE, even when not explicitly provided as text.
  • VQ-VAE as a "Phoneme Learner": The VQ-VAE effectively learns discrete representations that are sensitive to phonetic content, acting as an unsupervised "phoneme encoder."
  • Interpretability of Symbols: The learned symbolic representations can be visualized, showing that phonemes with similar pronunciations tend to activate similar symbol distributions. This provides a degree of interpretability to the model's internal workings.
  • Efficiency Gains: The proposed method achieved performance improvements with fewer trainable parameters compared to some baselines, partly due to the multi-head attention mechanism reducing dimensionality.

Key Results

  • Significant Performance Improvement: The proposed method consistently outperformed baseline models (U-Net and U-Net-MOL) in terms of both Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI) across various SNRs.
  • Outperforming Baselines: For example, at an average SNR, the proposed method achieved a PESQ score of 2.291 and STOI of 0.771, compared to U-Net (2.126 PESQ, 0.725 STOI) and U-Net-MOL (2.052 PESQ, 0.714 STOI). The "Oracle" model, which used ground-truth phoneme information, achieved the best scores, validating the benefit of linguistic input.
  • Optimal Symbolic Book Size: The performance peaked with a symbolic book size of 64, with larger sizes showing diminishing returns or issues like "index collapse."

Limitations and Future Work

  • Noise Sensitivity of Symbols: The authors acknowledge that noise can affect the input MFCCs, potentially confusing the symbolic encoder and leading to less accurate phonetic representations.
  • Noise-Invariant Training: To address the noise sensitivity, they suggest future work involving noise-invariant training, possibly using discriminators and adversarial training, to extract purer symbolic sequences.
  • Explicit Language Models: The paper proposes that incorporating an explicit language model constraint based on the learned symbols could further enhance SE performance.
  • Effect of Different Noise Types: Further investigation into how different noise types impact the symbolic encoder's performance is planned.

Summary updated

Abstract

In a noisy environment, a lossy speech signal can be automatically restored by a listener if he/she knows the language well.That is, with the built-in knowledge of a "language model", a listener may effectively suppress noise interference and retrieve the target speech signals.Accordingly, we argue that familiarity with the underlying linguistic content of spoken utterances benefits speech enhancement (SE) in noisy environments.In this study, in addition to the conventional modeling for learning the acoustic noisy-clean speech mapping, an abstract symbolic sequential modeling is incorporated into the SE framework.This symbolic sequential modeling can be regarded as a "linguistic constraint" in learning the acoustic noisy-clean speech mapping function.In this study, the symbolic sequences for acoustic signals are obtained as discrete representations with a Vector Quantized Variational Autoencoder algorithm.The obtained symbols are able to capture high-level phoneme-like content from speech signals.The experimental results demonstrate that the proposed framework can obtain notable performance improvement in terms of perceptual evaluation of speech quality (PESQ) and short-time objective intelligibility (STOI) on the TIMIT dataset.
BibTeX
@misc{liao2019incorporating,
  author = {Chien-Feng Liao and Yu Tsao and Xugang Lu and Hisashi Kawai},
  title = {{Incorporating Symbolic Sequential Modeling for Speech Enhancement}},
  year = {2019},
  eprint = {1904.13142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.13142},
}

Loading related content...

Figure