TL;DR
This paper tackles the challenge of real-time speech recognition with massive language models, which are too large for GPUs. They propose a hybrid GPU/CPU approach that leverages the GPU for initial decoding and the CPU for on-the-fly rescoring with a large language model, achieving an 11x speedup with comparable accuracy to traditional CPU-based methods.
What Problem Does This Paper Address?
Modern speech recognition systems, especially for tasks like voice search or broadcast news, rely on enormous acoustic and language models to achieve high accuracy. These models can contain millions of parameters and billions of n-gram contexts, making them computationally expensive and memory-intensive. While GPUs offer significant parallel processing power, their limited memory (1-6GB) makes it impossible to load these massive language models directly. This creates a bottleneck for real-time speech recognition, as traditional methods struggle to balance accuracy with speed when dealing with such large models. The paper aims to bridge this gap by efficiently integrating the computational power of GPUs with the larger memory capacity of CPUs.
Methods
The core of the proposed solution is a hybrid GPU/CPU architecture for Large Vocabulary Continuous Speech Recognition (LVCSR). The system uses a Weighted Finite-State Transducer (WFST) framework.
Here's a breakdown of their approach:
- GPU for Initial Search: The primary Viterbi search, which identifies potential word sequences, is performed on the GPU. This search uses a simplified WFST composed with a unigram language model. This keeps the WFST small enough to fit within the GPU's memory.
- CPU for On-the-Fly Rescoring: As the Viterbi search progresses on the GPU and encounters word boundaries, partial hypotheses (sequences of words) are sent to the CPU. On the CPU, these hypotheses are "rescored" using a much larger, higher-order language model (e.g., bigram or trigram).
- Hypothesis Likelihood Correction: The rescoring process involves calculating a correction term that represents the difference in likelihood between the unigram model used on the GPU and the higher-order model on the CPU. This correction is then applied to the hypothesis's score.
- N-Best Hypothesis Tracking: To mitigate the risk of pruning correct but less likely paths too early, the system maintains and tracks multiple (N-best) hypotheses at each state, rather than just the single best one. This allows the rescoring step on the CPU to have a richer set of candidates to evaluate.
- GPU Parallelism for N-Best Merging: A novel technique is introduced to efficiently merge these N-best lists on the GPU using atomic Compare-And-Swap operations, which are crucial for parallel processing in this context.
Datasets
The experiments were conducted on the Wall Street Journal (WSJ) task, specifically using the November 1992 ARPA WSJ test set (330 sentences). The acoustic model was trained on the Wall Street Journal SI-284 dataset. For language modeling, they used the standard WSJ 5k closed language models.
Key Insights and Ideas
- Hybridization is Key: The paper strongly advocates for a hybrid GPU/CPU approach as the most practical solution for handling massive language models in real-time speech recognition, leveraging the strengths of both architectures.
- Unigram WFST on GPU is Viable: Using a unigram language model to compose the WFST for GPU processing is an effective strategy to keep the model size manageable for the GPU, enabling fast initial search.
- Rescoring Bridges the Gap: On-the-fly rescoring on the CPU with a large language model is a powerful technique to recover accuracy lost by using a simplified model on the GPU.
- N-Best Tracking is Crucial: Simply using the single best hypothesis on the GPU and then rescoring is insufficient. Maintaining N-best hypotheses is essential to avoid premature pruning and capture more accurate paths.
- GPU Atomic Operations for Merging: The efficient merging of N-best lists on the GPU using atomic operations is a significant technical contribution for parallel processing in this domain.
Key Results
- Significant Speedup: The proposed hybrid GPU/CPU approach achieved an 11x speed-up compared to a standard CPU-based WFST decoding algorithm.
- Comparable Accuracy: The system achieved a word accuracy of 93.76% with the bigram rescoring, which is comparable to the standard bigram case.
- Improved Accuracy with Trigram: Using a trigram language model for rescoring further improved accuracy to 94.96%, exceeding the accuracy of a standard bigram WFST.
- N-Best Benefits: Maintaining just the 3-best paths during decoding provided most of the accuracy benefits of using a bigram language model.
- GPU Performance: The newer GTX680 GPU showed a 7% performance improvement over the GTX580, particularly in graph traversal and data gathering phases due to improved atomic operations.
Limitations and Future Work
- Rescoring Bottleneck: The paper notes that the on-the-fly rescoring phase, which runs on the CPU, can become a bottleneck, especially with larger N values.
- Future Work - Multi-core CPU Parallelism: To address the rescoring bottleneck, the authors intend to investigate using multi-core CPUs to parallelize this phase.
- Future Work - Adaptive Beamwidth: They also suggest further speed-up could be achieved by adaptively adjusting the search beam width based on the number of active sets.
Summary updated