Improving the Fusion of Acoustic and Text Representations in RNN-T

Chao Zhang, Bo Li, Zhiyun Lu, Tara N. Sainath, Shuo-Yiin Chang

TL;DR

Researchers at Google have optimized the "joint network"—the fusion hub of the Recurrent Neural Network Transducer (RNN-T)—to significantly improve streaming speech recognition. By replacing simple additive fusion with gating and bilinear pooling, and introducing a novel gradient-penalization trick to prevent the model from becoming "lazy" with audio features, they achieved a 4–5% relative reduction in Word Error Rate (WER) across a massive multilingual dataset.

The Bottleneck in RNN-T Architectures

In streaming Automatic Speech Recognition (ASR), the RNN-T is the current gold standard. It consists of three parts: an acoustic encoder (processing audio), a prediction network (acting as a language model for text), and a joint network that fuses these two streams to predict the next subword.

Historically, the joint network has been a simple layer that adds the two representations together. The authors argue this is a bottleneck for two reasons:

  1. Simple Fusion: Addition doesn't capture complex, multiplicative interactions between sound and text.
  2. The "Lazy" Encoder Problem: Text patterns are often easier to learn than complex acoustic patterns. Early in training, the model tends to rely too heavily on the prediction network (text priors), causing the acoustic encoder to underperform.

Advanced Fusion Structures

Instead of a basic fully connected layer, the authors propose several advanced structures for the joint network, as detailed in the table below.

Summary of proposed joint network structures and their parameter counts.

These structures include:

  • Gating Mechanism: Uses a sigmoid-activated gate to dynamically scale the importance of acoustic vs. text features for every single prediction.
  • Bilinear Pooling: Captures multiplicative interactions between every element of the acoustic and text vectors. To maintain efficiency, they use a low-rank approximation.
  • Stacked Combination (S6): Their best-performing architecture stacks bilinear pooling on top of a gated structure, leveraging the strengths of both.

Prediction Network Regularization

To solve the "lazy encoder" problem, the authors introduced a gradient-scaling factor ($\alpha_m$). During the early stages of training, they penalize the gradients flowing back into the prediction network. This forces the model to work harder on learning the acoustic features from the audio before it is allowed to rely fully on text-based language patterns. A piece-wise linear scheduler gradually "unleash" the prediction network as training progresses.

Experimental Results

The authors tested these methods on an industrial-scale dataset comprising 214.2 million anonymized utterances (~142,300 hours of speech) from Google’s Voice Search traffic across nine language locales.

Word Error Rate (WER) results across nine languages for various fusion structures and regularization techniques.

As shown in the table above, the combination of advanced fusion and gradient regularization (denoted with the "reg" superscript) led to a 4.2% to 5.1% relative WER reduction compared to the standard RNN-T baseline. These improvements were consistent across all nine languages, proving the robustness of the approach. Ablation studies showed that the regularization method alone improved the baseline WER by about 0.26% absolute.

Key Insights

  • Fusion Matters: Moving beyond simple addition to bilinear pooling allows the model to learn relationships between specific sounds and specific text contexts.
  • Balancing Modalities: By intentionally slowing down the "easier" text task, the "harder" audio task gets the gradient signal it needs to thrive.
  • Efficiency: While bilinear pooling sounds expensive, the low-rank approximation allows for a much more expressive model with only a tiny increase in total parameters (adding ~2.6M parameters to a 144M parameter model).

Limitations and Future Work

While the paper demonstrates clear gains, the impact on real-time factor (RTF) or latency remains to be detailed, which is critical for streaming ASR. Additionally, the gradient scheduler involves hyperparameters that might vary across different datasets. Future work could investigate if these fusion techniques provide similar boosts to other transducer-based models, like the Transformer-Transducer or the Emformer.

Summary updated

BibTeX
@inproceedings{zhang2022improving,
  author = {Chao Zhang and Bo Li and Zhiyun Lu and Tara N. Sainath and Shuo-Yiin Chang},
  title = {{Improving the Fusion of Acoustic and Text Representations in RNN-T}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747418},
}
Figure