Acoustic Modeling for Google Home.

Bo Li, Tara N. Sainath, Arun Narayanan, Joe Caroselli, Michiel Bacchiani, Ananya Misra, Izhak Shafran, Hasim Sak, Golan Pundak, Kean K. Chin, Khe Chai Sim, Ron J. Weiss, Kevin W. Wilson, Ehsan Variani, Chanwoo Kim, Olivier Siohan, Mitchel Weintraub, Erik McDermott, Richard Rose, Matt ShannonInterspeech 2017 · OpenReview

Abstract

This paper details the acoustic modeling innovations developed for the launch of Google Home. By integrating adaptive dereverberation (WPE), joint multichannel processing (fCLP), and frequency-aware Grid-LSTMs into a unified neural pipeline, the researchers achieved a 20% relative reduction in Word Error Rate (WER) compared to traditional speech recognition systems. The system effectively addresses the challenges of farfield recognition in noisy and reverberant home environments.

Introduction

Recognizing speech from across a room ("farfield" recognition) is significantly harder than recognizing speech into a handheld phone. In a typical home environment, the target voice is degraded by two main factors: reverberation (sound bouncing off walls) and additive noise (TVs, kitchen appliances, or music).

Traditionally, engineers solved this using a "divide and conquer" approach: a digital signal processing (DSP) frontend would clean the audio (beamforming and dereverberation), and a separate neural network would then try to recognize the words. This paper argues that these steps should be handled jointly. The challenge is building a model that is computationally efficient enough for real-time streaming while being robust enough to handle the messy acoustics of a living room.

Proposed Architecture

The authors propose a modular neural architecture that replaces standard "log-mel" audio features with a more sophisticated pipeline that processes raw Complex Fast Fourier Transform (CFFT) inputs.

The proposed neural architecture for farfield speech recognition, integrating WPE, fCLP, and Grid-LSTM modules.
  1. Weighted Prediction Error (WPE): An adaptive dereverberation frontend. It uses a multichannel recursive least squares (RLS) algorithm to estimate and subtract late reflections of sound in real-time, helping the network "see" through the echo.
  2. Factored Complex Linear Projection (fCLP): Instead of using pre-processed audio features, the model takes the CFFT as input. The fCLP layer mimics "filter-and-sum" beamforming within the network, allowing the model to learn how to spatially filter audio from multiple microphones.
  3. Grid-LSTM: Unlike standard LSTMs that only process data through time, a Grid-LSTM processes data across both time and frequency. The authors used a bidirectional version for the frequency axis, which helps the model better handle variations in pitch and vocal characteristics in noisy environments.
  4. LDNN (LSTM Deep Neural Network): The final stage is a standard deep architecture that predicts the actual phonetic targets.

Experimental Setup

  • Training Set: A massive 18,000-hour dataset (22 million utterances). This was created by taking clean voice search queries and "corrupting" them using a room simulator that added various levels of noise (sampled from YouTube) and simulated reverberation (RT60 up to 900ms).
  • Evaluation Sets: The models were tested on simulated data, "rerecorded" data (clean audio played back in a real room), and a "real" set consisting of 22,000 anonymized utterances from actual Google Home traffic.
  • Adaptation Set: 4,000 hours of real, live traffic data used for final "sequence training" to bridge the gap between simulation and reality.

Key Results and Insights

The "aha moment" of this research is that performing multichannel processing (beamforming) inside the neural network (via fCLP) outperforms traditional frontend processing. It allows the model to optimize the audio cleanup specifically for the task of recognition.

  • fCLP vs. Log-mel: The fCLP layer provided a 7% relative improvement in noisy conditions over standard features.
  • The Power of WPE: Adding the dereverberation frontend (WPE) provided an additional 7% improvement in the most reverberant settings.
  • Grid-LSTM Impact: This module improved performance by 7-11% across all test conditions.
Word Error Rate (WER) comparison between the production baseline and the proposed Google Home models across various noise types.

As shown in the results table, the final system reached an absolute WER of 4.9% on real Google Home traffic—a 20% relative reduction over the production baseline at the time. The researchers found that fine-tuning the model on just 4,000 hours of real traffic provided a massive boost (up to 28% relative WER improvement), highlighting the difficulty of perfectly simulating home acoustics.

Conclusion

This work demonstrates that joint learning of multichannel processing and acoustic modeling is highly effective for farfield recognition. While the WPE and Grid-LSTM components add computational overhead, they are essential for robustness in complex acoustic environments. Future directions include exploring joint architectures where every stage of the audio pipeline—from the raw waveform to the final transcript—is learned end-to-end.

Summary updated

BibTeX
@inproceedings{li2017acoustic,
  author = {Bo Li and Tara N. Sainath and Arun Narayanan and Joe Caroselli and Michiel Bacchiani and Ananya Misra and Izhak Shafran and Hasim Sak and Golan Pundak and Kean K. Chin and Khe Chai Sim and Ron J. Weiss and Kevin W. Wilson and Ehsan Variani and Chanwoo Kim and Olivier Siohan and Mitchel Weintraub and Erik McDermott and Richard Rose and Matt Shannon},
  title = {{Acoustic Modeling for Google Home.}},
  booktitle = {Interspeech},
  year = {2017},
  url = {https://openreview.net/forum?id=7K1Oz5RQ_z2},
}
Figure