Spatial Attention for Far-field Speech Recognition with Deep Beamforming Neural Networks

Weipeng He, Lu Lu, Biqiao Zhang, Jay Mahadeokar, Kaustubh Kalgaonkar, Christian Fuegen

Abstract

In this paper, we introduce spatial attention for refining the information in multi-direction neural beamformer for far-field automatic speech recognition. Previous approaches of neural beamformers with multiple look directions, such as the factored complex linear projection, have shown promising results. However, the features extracted by such methods contain redundant information, as only the direction of the target speech is relevant. We propose using a spatial attention subnet to weigh the features from different directions, so that the subsequent acoustic model could focus on the most relevant features for the speech recognition. Our experimental results show that spatial attention achieves up to 9% relative word error rate improvement over methods without the attention.

Introduction

Far-field Automatic Speech Recognition (ASR) is a critical component for smart home devices, yet it remains a significant challenge due to environmental factors like room reverberation, ambient noise, and competing speakers. Traditional beamforming techniques combine signals from a microphone array to enhance sound from a specific direction. Modern deep learning approaches, such as Factored Complex Linear Projection (fCLP), improve upon this by extracting features from multiple fixed "look directions" simultaneously.

While multi-directional features provide broad spatial coverage, they introduce a massive amount of redundant data. Since target speech typically originates from a single location, feeding all directional features into an Acoustic Model (AM) leads to parameter bloat and severe overfitting. This paper proposes a spatial attention mechanism to dynamically select and weight the most relevant directional features, effectively compressing the spatial information before it reaches the AM.

Methodology

The proposed architecture integrates a speech enhancement front-end with an ASR back-end in an end-to-end trainable framework.

Overall architecture of the proposed system featuring the spatial attention subnet.

The system consists of three primary stages:

  1. Neural Beamformer: An fCLP-based front-end takes raw multi-channel audio and applies $P$ spatial filters to extract spectral features ($Z_p$) for each look direction.
  2. Spatial Attention Subnet: A dedicated subnet, consisting of stacked Long Short-Term Memory (LSTM) layers, processes the multi-direction features to compute a spatial attention score $\mathbf{A}$ for each direction. These scores represent the relative importance of each spatial filter at any given time.
  3. Weighted Pooling: The final pooled feature $\bar{Z}$ is computed as a weighted average of the directional features: $\bar{Z} = \sum_{p=1}^{P} A_p Z_p$. This mechanism ensures that the downstream model receives a condensed, high-quality representation of the target speech.

The authors explored three temporal strategies for attention: Online (frame-by-frame), Offline (using the entire utterance), and Latency-Controlled (using a short initial segment, e.g., 1 second, to "lock" the attention for the rest of the sequence).

Experimental Setup

The models were trained on a massive dataset of 2 million utterances (~2,000 hours) of anonymized smart assistant commands. To simulate far-field conditions, clean audio was augmented with various room impulse responses and background noise. Evaluation was conducted on 16,500 utterances of real-world far-field audio recorded using Facebook Portal+ devices, which utilize a 4-microphone array.

Results and Discussion

The experimental results demonstrate that simply increasing the number of look directions without a selection mechanism is counterproductive. Without pooling, the multi-directional fCLP model (15.6% WER) actually performed worse than a simple single-channel log-mel baseline (14.3% WER) due to acoustic model overfitting.

Word Error Rate (WER) comparison across different front-end approaches and attention mechanisms.

As shown in the table, spatial attention consistently improves performance across all configurations. The online attention model reduced the WER to 13.0%. When combined with Weighted Prediction Error (WPE) dereverberation, the latency-controlled attention (1s) achieved a WER of 11.5%, representing a 9% relative improvement over the fCLP baseline.

Visual analysis of the attention weights revealed that the network typically requires about 2 seconds to confidently "lock on" to the target speaker's direction. The success of the latency-controlled variant is particularly significant for product design; it suggests that a brief initial audio segment—such as a wake word—is sufficient for the system to determine the optimal spatial focus for the subsequent command with zero added latency.

Conclusion

Spatial attention provides a robust solution to the redundancy problem in multi-direction neural beamformers. By dynamically weighting directional features, the model avoids parameter explosion and focuses on the most relevant acoustic information. The latency-controlled attention mechanism is particularly promising for production environments, where it can leverage wake-word detection to provide high-accuracy, low-latency speech recognition in challenging far-field conditions.

Summary updated

Abstract

In this paper, we introduce spatial attention for refining the information in multi-direction neural beamformer for far-field automatic speech recognition. Previous approaches of neural beamformers with multiple look directions, such as the factored complex linear projection, have shown promising results. However, the features extracted by such methods contain redundant information, as only the direction of the target speech is relevant. We propose using a spatial attention subnet to weigh the features from different directions, so that the subsequent acoustic model could focus on the most relevant features for the speech recognition. Our experimental results show that spatial attention achieves up to 9% relative word error rate improvement over methods without the attention.
BibTeX
@inproceedings{he2020spatial,
  author = {Weipeng He and Lu Lu and Biqiao Zhang and Jay Mahadeokar and Kaustubh Kalgaonkar and Christian Fuegen},
  title = {{Spatial Attention for Far-field Speech Recognition with Deep Beamforming Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=awQb3PL8qf},
}
Figure