Acoustic Modeling with Densely Connected Residual Network for Multichannel Speech Recognition

Jian Tang, Yan Song, Lirong Dai, Ian McLoughlinInterspeech 2018

TL;DR

This paper introduces DenseRNet, a novel deep learning architecture for multichannel speech recognition that combines the strengths of DenseNet and ResNet. By densely connecting residual network blocks, DenseRNet significantly improves robustness to noise and reverberation, achieving state-of-the-art results on the CHiME-3 dataset.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems often struggle in real-world scenarios characterized by background noise, reverberation, and distant speech. While microphone arrays and frontend signal processing techniques have been developed to mitigate these issues, the backend acoustic modeling component has received less attention. Existing models, like the simple DNN baseline used in the CHiME-3 challenge, are not always sufficient. This paper aims to improve the robustness and accuracy of the acoustic modeling part of multichannel ASR systems by proposing a new deep neural network architecture.

Methods

The core contribution of this paper is the DenseRNet architecture. It builds upon two successful deep learning concepts:

  • ResNet (Residual Networks): ResNets use "skip connections" or "residual connections" that allow gradients to flow more easily through deep networks, helping to overcome the vanishing gradient problem and enabling the training of much deeper models.
  • DenseNet (Densely Connected Networks): DenseNets connect each layer to every other layer in a feed-forward fashion within "dense blocks." This encourages feature reuse and strengthens gradient propagation.

DenseRNet merges these ideas by:

  • DenseR Blocks: These are the fundamental building blocks of DenseRNet. They are structured like ResNet's "building blocks" (containing convolutional layers) but are densely connected, similar to DenseNet. This means the output of each component within a denseR block is concatenated with the inputs of subsequent components.
  • Residual Transformation: Within the denseR block, a residual transformation is applied, combining the benefits of ResNet's skip connections with DenseNet's dense connectivity.
  • Multi-resolution Feature Exploitation: The dense connectivity allows the network to naturally exploit features at multiple resolutions.
  • Fully Convolutional Architecture: Unlike some previous approaches, DenseRNet is implemented as a fully convolutional network, avoiding the need for recurrent layers like BLSTM for temporal modeling.
  • Bottleneck Layers and Transition Layers: Similar to DenseNet, bottleneck layers (1x1 convolutions) are used to reduce feature map dimensions for computational efficiency, and transition layers are inserted between denseR blocks to further reduce dimensionality and improve compactness.

Datasets

The primary dataset used for training and evaluation in this paper is the CHiME-3 dataset. This dataset is known for its challenging real-world acoustic conditions, including noisy and reverberant speech recorded with microphone arrays. The authors also mention using specific subsets of the CHiME-3 data (CH5, Enh, CH0) to evaluate robustness.

Key Insights and Ideas

  • Synergy of DenseNet and ResNet: The paper demonstrates that combining the dense connectivity of DenseNet with the residual connections of ResNet leads to a more powerful and robust acoustic model than either architecture alone.
  • Improved Gradient Flow: The dense connections within denseR blocks significantly strengthen gradient back-propagation, effectively mitigating the vanishing gradient problem in deep networks.
  • Multi-resolution Feature Learning: The architecture naturally allows for the integration and exploitation of features extracted at different scales and resolutions, which is beneficial for handling diverse acoustic conditions.
  • Robustness to Noise and Reverberation: DenseRNet shows a remarkable ability to maintain performance across different acoustic conditions, including beamforming-enhanced speech, near-field, and far-field scenarios.
  • Parameter Efficiency: While achieving better performance, the DenseRNet models are shown to be relatively parameter-efficient compared to some other advanced architectures.

Key Results

  • State-of-the-Art Performance: DenseRNet achieved a Word Error Rate (WER) of 7.58% on the CHiME-3 real test data using beamforming-enhanced speech. This significantly outperforms the official baseline WER of 10.23%.
  • Robustness: The model demonstrated superior robustness compared to the baseline DNN across different input conditions (noisy far-field, beamforming-enhanced, and near-field speech).
  • Ablation Studies: Experiments showed that increasing the number of convolution layers (N) within the basic components of the denseR block and using a larger kernel size (S) generally improved performance, with the best results achieved by (N, S) = (3, 3). The growth rate (k) was also found to be important, with a non-fixed growth rate showing better results.

Limitations and Future Work

  • Parameter Tuning: The paper explores various parameter settings (L, k, N, S) for the denseR blocks, indicating that careful tuning is necessary for optimal performance. The authors pose questions about how to best set these parameters, suggesting further research is needed.
  • Computational Cost: While aiming for efficiency, deep learning models can still be computationally intensive. The paper mentions using bottleneck layers to reduce computational load, but further optimization might be explored.
  • Generalizability: While successful on CHiME-3, the generalizability of DenseRNet to other challenging ASR datasets and acoustic environments could be a direction for future work.
  • Further Exploration of DenseR Block Design: The paper suggests that the design of the "building blocks" within the denseR block could be further investigated for even greater improvements.

Summary updated

BibTeX
@inproceedings{tang2018acoustic,
  author = {Jian Tang and Yan Song and Lirong Dai and Ian McLoughlin},
  title = {{Acoustic Modeling with Densely Connected Residual Network for Multichannel Speech Recognition}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure