Quaternion Neural Networks for Multi-channel Distant Speech Recognition

Xinchi Qiu, Titouan Parcollet, Mirco Ravanelli, Nicholas D. Lane, Mohamed Morchid

Abstract

Despite the significant progress in automatic speech recognition (ASR), distant ASR remains challenging due to noise and reverberation. A common approach to mitigate this issue consists of equipping the recording devices with multiple microphones that capture the acoustic scene from different perspectives. These multi-channel audio recordings contain specific internal relations between each signal. In this paper, we propose to capture these inter- and intra- structural dependencies with quaternion neural networks, which can jointly process multiple signals as whole quaternion entities. The quaternion algebra replaces the standard dot product with the Hamilton one, thus offering a simple and elegant way to model dependencies between elements. The quaternion layers are then coupled with a recurrent neural network, which can learn long-term dependencies in the time domain. We show that a quaternion long-short term memory neural network (QLSTM), trained on the concatenated multi-channel speech signals, outperforms equivalent real-valued LSTM on two different tasks of multi-channel distant speech recognition.

Introduction

Automatic Speech Recognition (ASR) has achieved remarkable success in "close-talking" scenarios. However, performance degrades significantly in distant-talking environments where background noise and room reverberation interfere with the signal. To address this, multi-microphone arrays are frequently employed. Traditional methods either use beamforming to spatially filter the audio or concatenate all channel data into a high-dimensional vector for a standard neural network. The latter approach often fails to capture the inherent spatial relationships between microphones efficiently, requiring a massive number of parameters to learn these dependencies.

This paper introduces Quaternion Long Short-Term Memory (QLSTM) networks. By leveraging quaternion algebra, the model treats multi-channel signals as a single multidimensional entity, capturing complex inter-channel dependencies more effectively than standard real-valued LSTMs while maintaining parameter efficiency.

Methodology: Quaternion Neural Networks

The core innovation lies in using Quaternion Neural Networks (QNNs) to provide a better inductive bias for multi-channel data. Instead of standard real numbers, the network operates in the quaternion domain ($Q = a + bi + cj + dk$).

The Hamilton Product

In a standard neural network, layers rely on dot products. QNNs replace these with the Hamilton product. This operation enforces a specific weight-sharing structure across the four dimensions, which is ideal for modeling the "inter- and intra-microphone" dependencies.

Quaternion input mapping and Hamilton product operation.

QLSTM Architecture

The researchers developed a Recurrent Neural Network (RNN) where every gate (forget, input, output) and weight matrix is quaternion-valued. Audio features (such as MFCCs or Filterbanks) from four distinct microphones are mapped directly onto the four components ($a, b, c, d$) of the quaternion input $Q_{in}$.

Experimental Setup

The models were evaluated on two benchmarks:

  1. Multi-channel TIMIT: A simulated dataset created by applying impulse responses from a real living room to simulate four ceiling-mounted microphones.
  2. DIRHA Corpus: A challenging dataset featuring real and simulated recordings in a domestic environment with significant non-stationary noise (e.g., kitchen sounds) and high reverberation.

Results and Discussion

The QLSTM consistently outperformed both standard LSTMs and traditional beamforming techniques across different feature sets.

Performance comparison (Phoneme Error Rate) on the multi-channel TIMIT dataset.

Key Findings

  • Structural Weight Sharing: The Hamilton product acts as a form of regularization, forcing the network to learn how changes in one microphone signal relate to others.
  • Parameter Efficiency: A quaternion layer requires 4x fewer parameters than a real-valued layer of equivalent width. This allows for deeper or more efficient models.
  • Superiority over Beamforming: The QLSTM achieved a 28.7% Phoneme Error Rate (PER) on TIMIT, outperforming beamforming (31.1%) and standard LSTMs (30.2%). On the real DIRHA dataset, it reached a 29.8% Word Error Rate (WER), compared to 32.7% for the LSTM baseline.

Conclusion and Future Work

While QLSTMs show great promise, they are currently limited to four-dimensional inputs. Future work may explore Octonions (8D) or Sedenions (16D) to accommodate devices with more microphones. Additionally, addressing the computational overhead of the Hamilton product and developing efficient quaternion-based normalization techniques remain key areas for improvement.

Summary updated

Abstract

Despite the significant progress in automatic speech recognition (ASR), distant ASR remains challenging due to noise and reverberation. A common approach to mitigate this issue consists of equipping the recording devices with multiple microphones that capture the acoustic scene from different perspectives. These multi-channel audio recordings contain specific internal relations between each signal. In this paper, we propose to capture these inter- and intra- structural dependencies with quaternion neural networks, which can jointly process multiple signals as whole quaternion entities. The quaternion algebra replaces the standard dot product with the Hamilton one, thus offering a simple and elegant way to model dependencies between elements. The quaternion layers are then coupled with a recurrent neural network, which can learn long-term dependencies in the time domain. We show that a quaternion long-short term memory neural network (QLSTM), trained on the concatenated multi-channel speech signals, outperforms equivalent real-valued LSTM on two different tasks of multi-channel distant speech recognition.
BibTeX
@inproceedings{qiu2020quaternion,
  author = {Xinchi Qiu and Titouan Parcollet and Mirco Ravanelli and Nicholas D. Lane and Mohamed Morchid},
  title = {{Quaternion Neural Networks for Multi-channel Distant Speech Recognition}},
  booktitle = {Interspeech},
  year = {2020},
  url = {https://openreview.net/forum?id=jK97gSLGxRU},
}
Figure