West: Word Encoded Sequence Transducers.

Ehsan Variani, Ananda Theertha Suresh, Mitchel Weintraub

Introduction

In the fields of Natural Language Processing (NLP) and Automatic Speech Recognition (ASR), large vocabulary models often face a significant memory "tax." While the core logic of these models resides in recurrent layers or transformers, the majority of the parameter budget is typically consumed by the "dictionary"—the massive embedding and softmax layers that map words to vectors and vice versa.

The Parameter Bottleneck

In a typical ASR system, the embedding (input) and softmax (output) layers can account for over 90% of the total parameters. This overhead is a critical barrier for on-device inference on mobile hardware with limited RAM and for federated learning, where transmitting massive updates is impractical.

Baseline parameter distribution in a typical large-vocabulary model, where Embedding and Softmax layers dominate.

The WEST Method

Google researchers introduced WEST (Word Encoded Sequence Transducers) to address this bottleneck. WEST is a single-stage compression technique that factors massive word-level matrices into the product of a structured sparse matrix ($C$) and a dense sub-unit matrix ($E^c$): $E = C \times E^c$.

By representing words as sequences of sub-units—such as characters or even random codes—WEST achieves up to 1000x compression of these layers. Crucially, the model maintains a word-level cycle, normalizing probabilities at the word step rather than the sub-unit step. This "MaxEnt" approach preserves the modeling benefits of word-level representations while utilizing the memory footprint of character-level models.

Reallocation and Performance

One of the most significant advantages of WEST is the ability to reallocate saved memory. By shrinking the "dictionary" layers, researchers can expand the "brains" of the model—the LSTM or recurrent layers.

Evolution of parameter distribution from baseline to WEST-compressed models, showing the reallocation of memory to the LSTM layers.

In large-scale ASR tasks, this reallocation strategy allowed for a reduction in total model size from 15MB to 4.75MB. When combined with scalar quantization, the model reached 1.35MB (an 11x reduction) while actually improving the Word Error Rate (WER) by 3% relative to the baseline.

Conclusion

WEST demonstrates that linguistic characters are not the only way to compress models; random coding can be equally effective. By focusing on word-level normalization and strategic parameter reallocation, WEST enables the deployment of high-performance, large-vocabulary models on resource-constrained devices without sacrificing accuracy.

Reference: Variani, E., Suresh, A. T., & Weintraub, M. (2018). WEST: Word Encoded Sequence Transducers. arXiv:1811.08417v1 [cs.LG].

Summary updated

Abstract

Most of the parameters in large vocabulary models are used in embedding layer to map categorical features to vectors and in softmax layer for classification weights. This is a bottleneck in memory constraint on-device training applications like federated learning and on-device inference applications like automatic speech recognition (ASR). One way of compressing the embedding and softmax layers is to substitute larger units such as words with smaller sub-units such as characters. However, often the sub-unit models perform poorly compared to the larger unit models. We propose WEST, an algorithm for encoding categorical features and output classes with a sequence of random or domain dependent sub-units and demonstrate that this transduction can lead to significant compression without compromising performance.
BibTeX
@misc{variani2018west,
  author = {Ehsan Variani and Ananda Theertha Suresh and Mitchel Weintraub},
  title = {{West: Word Encoded Sequence Transducers.}},
  year = {2018},
  eprint = {1811.08417},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.08417},
}
Figure