Spatial Attention for Far-Field Speech Recognition with Deep Beamforming Neural Networks

Weipeng He, Lu Lu, Biqiao Zhang, Jay Mahadeokar, Kaustubh Kalgaonkar, Christian Fuegen

TL;DR

This paper introduces a spatial attention mechanism for multi-direction deep neural beamformers to improve far-field Automatic Speech Recognition (ASR). By dynamically weighing the features from different spatial "look directions," the model filters out redundant information, reduces overfitting, and achieves up to a 9% relative improvement in Word Error Rate (WER) over baseline multi-directional methods.

What Problem Does This Paper Address?

Far-field ASR—the technology powering smart speakers across a room—is notoriously difficult. Unlike near-field speech (like talking directly into a phone), far-field audio is heavily corrupted by room reverberation, background noise, and interfering voices.

To isolate the target speaker, systems use beamforming, which combines signals from a microphone array to enhance sound coming from a specific direction. Recently, deep learning approaches like the Factored Complex Linear Projection (fCLP) have shown great promise by using neural networks to extract features from multiple fixed look directions simultaneously. However, feeding the features from all these directions into the downstream Acoustic Model (AM) creates a massive, redundant feature space. Since the target speech only comes from one (or a few) directions, forcing the AM to process all directions increases the parameter count and makes the network highly prone to overfitting.

Methods

The authors propose an end-to-end neural network that jointly trains a speech enhancement front-end and an ASR back-end. The architecture has three main components:

  1. Neural Beamformer: Based on fCLP, this front-end takes raw multi-channel audio and applies $P$ spatial filters (look directions) to extract spectral features for each direction.
  2. Attention-Based Pooling Module (The Novelty): Instead of concatenating all directional features, a dedicated subnet (built with stacked LSTMs) calculates a "spatial attention" score for each direction. These scores are used to compute a weighted average of the features across all directions, passing only the most informative, condensed representation forward.
  3. Acoustic Model Back-End: A standard 5-layer LSTM that predicts the speech sequence from the pooled features.

The authors experimented with three ways to compute this attention:

  • Online: Frame-by-frame attention prediction (smoothed over time).
  • Offline: Computes attention using the entire utterance, applying the final frame's score to the whole sequence (highly accurate, but introduces latency).
  • Latency-Controlled: Uses only a short initial segment of the audio (e.g., 0.5 or 1 second) to predict the attention scores for the rest of the sequence, mimicking how a smart speaker might use a "wake word" to lock onto a spatial direction.

Datasets

  • Training: 2 million utterances (~2,000 hours) from an in-house, anonymized crowdsourced dataset of smart assistant commands. These were artificially corrupted with simulated room impulse responses (reverberation) and background noise to mimic far-field conditions.
  • Evaluation: 16,500 utterances (~16 hours) of real-world far-field audio. This was collected by playing clean speech and noise through loudspeakers and recording the audio using Facebook Portal+ devices (which feature a 4-microphone rectangular array).

Key Insights and Ideas

  • Attention prevents parameter bloat: Simply concatenating features from all directions (the baseline fCLP approach) actually performed worse than a standard single-channel log-mel baseline. The model had too many parameters and overfit. Spatial attention solves this by dynamically compressing the spatial dimension.
  • Spatial filters are multi-taskers: When visualizing the directivity patterns of the learned beamformers, the authors found that individual filters didn't just point in one single direction. Because the network only uses 10 filters to cover a 3D space, each filter learns to amplify multiple directions simultaneously to maximize coverage.
  • Attention takes time to focus: Visualizing the attention scores over time revealed that the network spends the first ~2 seconds of an utterance distributing its attention evenly across all directions. Only after this initial period does it confidently "lock on" to a few specific spatial filters.
  • Wake words are perfect for spatial locking: The success of the "latency-controlled" attention variant proves that a system only needs a brief snippet of audio to figure out where the user is standing. In a real-world product, the wake word (e.g., "Hey Portal") could be used to compute the spatial attention weights, allowing the actual command to be processed with zero added latency.

Key Results

  • Baseline Performance: Standard log-mel features with no enhancement yielded a 14.3% WER. The fCLP model without pooling (concatenating all directions) degraded to 15.6% WER.
  • Online Attention: Adding online spatial attention improved the WER to 13.0% (a significant jump over the no-pooling baseline).
  • Best Practical Setup: Combining Weighted Prediction Error (WPE) dereverberation with latency-controlled attention (1s) achieved an 11.5% WER. This represents a 9% relative improvement over the baseline fCLP model.
  • Theoretical Best: Offline attention (which "cheats" by looking at the whole utterance before deciding on a direction) achieved the lowest error rate at 11.3% WER.

Limitations and Future Work

  • Latency Trade-offs: The highest-performing offline attention model cannot be used for real-time streaming ASR due to latency. While latency-controlled attention mitigates this, it still requires a buffer.
  • Diminishing Returns on Filters: The authors noted that increasing the number of look directions from 10 to 20 did not improve the WER, as the attention mechanism simply saturated.
  • Future Work: The authors strongly suggest integrating this approach directly with wake-word detection systems, using the trigger phrase exclusively to predict the spatial attention weights for the subsequent command.

Summary updated

Abstract

In this paper, we introduce spatial attention for refining the information in multi-direction neural beamformer for far-field automatic speech recognition. Previous approaches of neural beamformers with multiple look directions, such as the factored complex linear projection, have shown promising results. However, the features extracted by such methods contain redundant information, as only the direction of the target speech is relevant. We propose using a spatial attention subnet to weigh the features from different directions, so that the subsequent acoustic model could focus on the most relevant features for the speech recognition. Our experimental results show that spatial attention achieves up to 9% relative word error rate improvement over methods without the attention.
BibTeX
@inproceedings{he2020spatial,
  author = {Weipeng He and Lu Lu and Biqiao Zhang and Jay Mahadeokar and Kaustubh Kalgaonkar and Christian Fuegen},
  title = {{Spatial Attention for Far-Field Speech Recognition with Deep Beamforming Neural Networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=awQb3PL8qf},
}

Loading related content...

Figure