Echo State Speech Recognition

Harsh Shrivastava, Ankush Garg, Yuan Cao, Yu Zhang, Tara Sainath

TL;DR

Modern Automatic Speech Recognition (ASR) models are notoriously parameter-heavy and expensive to train. This paper demonstrates that you don't actually need to train the whole thing: by replacing the recurrent decoder layers with random, untrained Echo State Networks (ESNs), researchers maintained (and sometimes improved) accuracy while slashing training time by up to 37% and significantly reducing model storage requirements.

What Problem Does This Paper Address?

In the world of ASR, the standard operating procedure is to train every single parameter of a model—often numbering in the hundreds of millions—using backpropagation. This "heavy-lifting" approach is computationally expensive and assumes that every component requires fine-tuning to function.

The authors challenge this assumption by investigating the sensitivity of different model components. Specifically, they ask: Is end-to-end training truly necessary for every layer? By using Echo State Networks—a type of "reservoir computing" where recurrent connections are random and fixed—they aim to identify which parts of an ASR system are doing the hard work of representation learning and which are simply managing temporal dynamics.

Methods

The researchers integrated ESNs into two state-of-the-art architectures: RNN-Transducer (RNN-T) and Conformer.

Standard ESNs consist of a "reservoir" (a recurrent layer) with random, fixed weights and a trainable "readout" layer. The authors adapted this by:

  1. Replacing RNN layers: They swapped out trainable LSTM layers in the encoder or decoder for ESN layers.
  2. Learnable Scaling: Instead of manual hyperparameter tuning, they introduced two learnable scalars—spectral radius ($\rho$) and input scale ($\gamma$)—per layer. This allows backpropagation to optimize the global dynamics of the random reservoir without ever touching the individual random weights.
  3. Sparsity: They set the ESN recurrent matrices to 80% sparsity, meaning only 20% of the random connections were non-zero.
Fig. 1. Architecture of RNN-T (left) and Conformer encoder (right).

Datasets

The models were trained and evaluated on:

  • LibriSpeech: A standard corpus of 970 hours of labeled speech. Results were reported on the "test clean" and "test other" (more challenging) splits.
  • Long-form Dataset: To test stability over time, the authors created a custom set of 98 examples by concatenating utterances, resulting in audio clips between 100 and 350 seconds long.

Key Insights and Ideas

  • The Decoder is "Easy": The most surprising finding is that randomizing the decoder doesn't hurt performance. In fact, for RNN-T, the ESN decoder actually outperformed the fully trained baseline on long-form audio. This suggests that the decoder's role is primarily to track state, a task a random reservoir handles naturally.
  • The Encoder is the "Brain": Conversely, randomizing the encoder caused Word Error Rates (WER) to skyrocket (e.g., from 6.9% to 35.6%). This confirms that the encoder performs the critical task of learning complex acoustic representations that cannot be captured by random chance.
  • Seed-Based Storage: Because ESN weights are random but deterministic, you don't need to store the actual weight matrices. You only need to store a single random seed. For an RNN-T model, this could shrink the decoder's storage footprint from millions of parameters down to a single integer.

Key Results

The quantitative results, summarized in the table below, highlight the efficiency of the ESN-decoder approach:

WER comparison for different models
  • Accuracy: The 512-dimension RNNT-D (random decoder) achieved a 6.3% WER on test-clean, beating the fully trained baseline's 6.8%.
  • Training Speed: The RNNT-D model trained 37% faster than the baseline (3.5 hours vs 5.5 hours to reach 10k steps) because no gradients were calculated for the decoder layers.
  • Long-form Stability: On long-form examples, the ESN-based RNN-T reduced WER from 18.1% to 17.5%, suggesting better-conditioned gradient flow during training for long sequences.

Limitations and Future Work

While the results for RNN-T were dramatic, the Conformer model saw less benefit. Because 97% of a Conformer's parameters reside in its encoder—which cannot be randomized without massive quality loss—the training speedup was negligible.

The authors also noted that while their ESN matrices are 80% sparse, current hardware often doesn't provide significant speedups for sparse operations. Future work could involve optimizing hardware kernels to fully exploit this sparsity for even faster inference on mobile devices. Additionally, exploring how this "randomized" approach applies to non-recurrent components, like pure Transformer blocks, remains an open question.

Summary updated

Abstract

We propose automatic speech recognition (ASR) models inspired by echo state network (ESN), in which a subset of recurrent neural networks (RNN) layers in the models are randomly initialized and untrained. Our study focuses on RNN-T and Conformer models, and we show that model quality does not drop even when the decoder is fully randomized. Furthermore, such models can be trained more efficiently as the decoders do not require to be updated. By contrast, randomizing encoders hurts model quality, indicating that optimizing encoders and learn proper representations for acoustic inputs are more vital for speech recognition. Overall, we challenge the common practice of training ASR models for all components, and demonstrate that ESN-based models can perform equally well but enable more efficient training and storage than fully-trainable counterparts.
BibTeX
@inproceedings{shrivastava2021echo,
  author = {Harsh Shrivastava and Ankush Garg and Yuan Cao and Yu Zhang and Tara Sainath},
  title = {{Echo State Speech Recognition}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ShLlfy5qvl},
}

Loading related content...

Figure