Abstract
This paper presents a highly compressed Recurrent Neural Network Language Model (RNN-LM) designed for mobile keyboards. By integrating knowledge distillation with a novel shared matrix factorization technique, the authors developed a model that fits within a 7.4MB memory footprint and achieves a response time under 7ms. Experimental results demonstrate that this approach outperforms major commercial solutions in keystroke savings and word prediction accuracy.
Introduction
Recurrent Neural Networks (RNNs) are effective for next-word prediction but typically require substantial memory (over 50MB), exceeding the constraints of mobile devices (~10MB). Furthermore, real-time user experience requires prediction latency below 10ms. Traditional compression methods like pruning often result in irregular memory access, hindering execution speed. This work addresses the challenge of reducing model size by 8x while maintaining high accuracy and low latency.
Methodology
The researchers employed a multi-stage compression pipeline to squeeze the model without sacrificing performance:
- Knowledge Distillation (KD): A small "student" model is trained using an ensemble of large "teacher" models to learn the underlying language probability distribution more effectively than training on raw data alone.
- Shared Matrix Factorization (SMF): Recognizing the semantic symmetry between input (embedding) and output (softmax) layers, the authors forced these layers to share a single weight matrix ($W_{shared}$). Small projection matrices adapt this shared matrix for specific layer needs, effectively cutting the parameter count of these layers in half.
- Low-Rank Approximation (SVD): Singular Value Decomposition is used to initialize the factored matrices, ensuring the compressed model starts from a mathematically sound approximation of the original.
- Quantization: Weights are converted from 32-bit floating-point to 16-bit integers, further reducing storage with minimal accuracy loss.
The use of matrix factorization ensures the matrices remain "dense," allowing the use of standard, optimized hardware libraries to maintain a fast response time of 6.47ms, unlike sparse pruning methods.
Evaluation
The model was trained on a 196-million-word sample from social network data and evaluated on a benchmark of 102 sentences representing diverse typing scenarios. The final model achieved a size reduction from 56.76MB to 7.40MB.
Performance was benchmarked against leading commercial keyboards on a Samsung Galaxy S7. The primary metrics used were Keystroke Savings (KS), measuring the percentage of characters a user doesn't have to type, and Word Prediction Rate (WPR), the frequency of correct word predictions.
As shown in the results, the proposed model achieved a KS of 65.11% and a WPR of 34.38%, surpassing industry standards including iOS, Swiftkey, and G-board.
Conclusion
The study demonstrates that combining knowledge distillation with hardware-friendly matrix factorization allows for state-of-the-art neural language models to operate efficiently on-device. Future work includes expanding the vocabulary beyond 15,000 words, implementing on-device personalization, and extending the methodology to morphologically rich languages.
Summary updated