TL;DR
This paper tackles the challenge of building a single, efficient language model for on-device Automatic Speech Recognition (ASR) that can handle diverse user requests (like virtual assistants and dictation) without sacrificing accuracy or speed. They propose novel "Application-Agnostic" and "Application-Dependent" feed-forward architectures and a data balancing strategy to achieve this, significantly reducing model size compared to separate application-specific models.
What Problem Does This Paper Address?
On-device ASR systems, like those in smartphones, face a tough balancing act: they need to be fast, memory-efficient, and small in storage size, all while delivering accurate speech recognition. A major hurdle is serving multiple applications with distinct linguistic patterns. For instance, a virtual assistant (VA) might handle short, command-based queries ("play music"), while speech-to-text (STT) deals with longer, more narrative dictations. The most accurate solution is to train separate language models for each application, but this drastically increases the model's memory footprint, making it impractical for resource-constrained devices. This paper aims to bridge this gap by developing a single, unified language model that can effectively serve both VA and STT use cases.
Methods
The authors introduce two novel feed-forward neural network language model (NNLM) architectures based on the Fixed-size Ordinally-Forgetting Encoding (FOFE) approach:
- Application-Agnostic (AA) Mixture NNLM: This architecture uses an ensemble of parallel sub-networks, each potentially specialized, and a "mixture sub-network." The mixture sub-network learns to dynamically combine the outputs of the parallel sub-networks using learned weights, effectively creating a single model that adapts to different applications.
- Application-Dependent (AD) FOFE NNLM: This approach leverages application information during training. The model has separate sub-networks and output biases for each application. During training, gradients are only back-propagated through the relevant sub-network. At inference, the system uses the application context to select the appropriate sub-network.
Crucially, they also propose a data balancing scheme. This method aims to optimally sample data from different sources (VA and STT) to create a unified training dataset. It involves calculating "redistribution probability masses" for each application based on their performance on a balanced development set, and then using these to re-normalize sampling weights for the training data.
Datasets
The paper uses anonymized and randomly sampled user requests from both VA and STT applications for training and evaluation. They also incorporate synthetic "tail-focused" datasets, which are generated from domain-dependent templates and entity lists to cover less frequent queries. The evaluation is performed on three types of test sets: VA, STT, and a "Tail" set designed to focus on queries with less common entities. These tests are conducted across three languages: US English, German, and Mandarin Chinese.
Key Insights and Ideas
- The "one-size-fits-all" challenge: The paper highlights that simply merging data from different applications into a single model can lead to accuracy degradation. This underscores the need for intelligent data balancing and architectural innovations.
- Architecture matters for efficiency: The proposed AA Mixture and AD FOFE architectures demonstrate that it's possible to achieve good performance without the prohibitive memory cost of separate models. The AD FOFE, in particular, shows promise for balancing accuracy and speed.
- Data balancing is critical: The proposed data balancing scheme is key to making a single model work. By intelligently weighting data sources, they can mitigate the negative effects of combining diverse linguistic patterns.
- FOFE's resilience: The paper suggests that FOFE-based models, even with short context lengths, can be competitive with more complex architectures like Transformers, especially in resource-constrained environments.
Key Results
- The AD FOFE model achieved the best accuracy across most test sets and languages, often outperforming application-specific models and Transformers.
- The AA Mixture FOFE also showed strong performance, particularly on STT tasks.
- Compared to application-specific FOFE models, the AA FOFE models (which have roughly half the parameters) showed a WER degradation of 1.5-3.8% in some cases, demonstrating a good trade-off between size and accuracy.
- The AD FOFE model offered the fastest inference speed among the proposed models, even outperforming the baseline FOFE on English VA and all German test sets.
- The Transformer models, while competitive in some English scenarios, showed significant accuracy degradations on other language/task combinations.
Limitations and Future Work
- Limited Transformer Comparison: The paper notes that while they compared their models to Transformer models, this comparison was not exhaustive across all languages and test sets.
- Word-Level LMs: The current work focuses on word-level language models. Preliminary experiments with subword-level LMs have been conducted, suggesting that further investigation in this area could yield more insights.
- Reproducibility: The authors acknowledge that exact reproduction of their setup is difficult due to the lack of public datasets that precisely mimic their ASR system, large-scale data, and specific test set configurations.
- Future Directions: The paper implicitly suggests exploring subword modeling and further optimizing the balance between accuracy, latency, and model size for on-device ASR.
Summary updated