Abstract
Despite the significant progress in automatic speech recognition (ASR), distant ASR remains challenging due to noise and reverberation. A common approach to mitigate this issue consists of equipping the recording devices with multiple microphones that capture the acoustic scene from different perspectives. These multi-channel audio recordings contain specific internal relations between each signal. In this paper, we propose to capture these inter- and intra- structural dependencies with quaternion neural networks, which can jointly process multiple signals as whole quaternion entities. The quaternion algebra replaces the standard dot product with the Hamilton one, thus offering a simple and elegant way to model dependencies between elements. The quaternion layers are then coupled with a recurrent neural network, which can learn long-term dependencies in the time domain. We show that a quaternion long-short term memory neural network (QLSTM), trained on the concatenated multi-channel speech signals, outperforms equivalent real-valued LSTM on two different tasks of multi-channel distant speech recognition.
Introduction
Automatic Speech Recognition (ASR) has achieved remarkable success in "close-talking" scenarios. However, performance degrades significantly in distant-talking environments where background noise and room reverberation interfere with the signal. To address this, multi-microphone arrays are frequently employed. Traditional methods either use beamforming to spatially filter the audio or concatenate all channel data into a high-dimensional vector for a standard neural network. The latter approach often fails to capture the inherent spatial relationships between microphones efficiently, requiring a massive number of parameters to learn these dependencies.
This paper introduces Quaternion Long Short-Term Memory (QLSTM) networks. By leveraging quaternion algebra, the model treats multi-channel signals as a single multidimensional entity, capturing complex inter-channel dependencies more effectively than standard real-valued LSTMs while maintaining parameter efficiency.
Methodology: Quaternion Neural Networks
The core innovation lies in using Quaternion Neural Networks (QNNs) to provide a better inductive bias for multi-channel data. Instead of standard real numbers, the network operates in the quaternion domain ($Q = a + bi + cj + dk$).
The Hamilton Product
In a standard neural network, layers rely on dot products. QNNs replace these with the Hamilton product. This operation enforces a specific weight-sharing structure across the four dimensions, which is ideal for modeling the "inter- and intra-microphone" dependencies.
QLSTM Architecture
The researchers developed a Recurrent Neural Network (RNN) where every gate (forget, input, output) and weight matrix is quaternion-valued. Audio features (such as MFCCs or Filterbanks) from four distinct microphones are mapped directly onto the four components ($a, b, c, d$) of the quaternion input $Q_{in}$.
Experimental Setup
The models were evaluated on two benchmarks:
- Multi-channel TIMIT: A simulated dataset created by applying impulse responses from a real living room to simulate four ceiling-mounted microphones.
- DIRHA Corpus: A challenging dataset featuring real and simulated recordings in a domestic environment with significant non-stationary noise (e.g., kitchen sounds) and high reverberation.
Results and Discussion
The QLSTM consistently outperformed both standard LSTMs and traditional beamforming techniques across different feature sets.
Key Findings
- Structural Weight Sharing: The Hamilton product acts as a form of regularization, forcing the network to learn how changes in one microphone signal relate to others.
- Parameter Efficiency: A quaternion layer requires 4x fewer parameters than a real-valued layer of equivalent width. This allows for deeper or more efficient models.
- Superiority over Beamforming: The QLSTM achieved a 28.7% Phoneme Error Rate (PER) on TIMIT, outperforming beamforming (31.1%) and standard LSTMs (30.2%). On the real DIRHA dataset, it reached a 29.8% Word Error Rate (WER), compared to 32.7% for the LSTM baseline.
Conclusion and Future Work
While QLSTMs show great promise, they are currently limited to four-dimensional inputs. Future work may explore Octonions (8D) or Sedenions (16D) to accommodate devices with more microphones. Additionally, addressing the computational overhead of the Hamilton product and developing efficient quaternion-based normalization techniques remain key areas for improvement.
Summary updated