TL;DR
This paper presents a novel method for significantly speeding up speech recognition by offloading acoustic likelihood computations to Graphics Processing Units (GPUs). They demonstrate that a hierarchical approach, which intelligently prunes unnecessary calculations, can be effectively implemented on GPUs, achieving a 2x speedup over a full Gaussian computation on the GPU and a substantial improvement over CPU-based methods.
What Problem Does This Paper Address?
Speech recognition systems are computationally intensive, with a significant portion of the processing time (30-50%) dedicated to calculating acoustic likelihoods. Traditional methods for optimizing this, like hierarchical Gaussian selection, work well on CPUs by reducing the number of calculations. However, these pruning techniques are challenging to implement efficiently on GPUs due to their parallel processing architecture and memory access patterns. This paper aims to bridge this gap by demonstrating how to adapt hierarchical Gaussian selection for GPU acceleration, offering a more efficient alternative to both CPU-based hierarchical methods and simpler, but less efficient, full Gaussian computations on GPUs.
Methods
The authors propose a heterogeneous approach, leveraging both CPUs and GPUs for different tasks within the hierarchical acoustic likelihood computation.
- Full Gaussian Computation (Baseline): This involves computing the likelihood for every Gaussian component in the model. This is a natural fit for GPUs due to its matrix multiplication nature.
- Hierarchical Gaussian Selection: This method first clusters Gaussians and then evaluates only the clusters deemed "active" (likely to contribute to the observation). This significantly reduces the number of computations.
- GPU Implementation of Hierarchical Method:
- Cluster Likelihoods: Computed on the GPU using the full Gaussian evaluation.
- Active Cluster Identification: Performed on the CPU using an "n-th element" algorithm to select the top-scoring clusters.
- Active Gaussian Evaluation: The likelihoods of Gaussians within active clusters are computed on the GPU. A key innovation here is reordering the data in GPU memory to group Gaussians by cluster for efficient parallel processing, even though this is less ideal for the subsequent mixture likelihood computation.
- Mixture Likelihoods: Evaluated on the GPU using a reduction algorithm.
- Optimization Strategies: The paper emphasizes the importance of work-group and cluster sizes, and processing multiple frames in a single GPU kernel call to maximize efficiency. They also discuss techniques to avoid "inactive" work-items altogether, further boosting performance.
Datasets
The experiments were conducted using the EARS'04 development speech data, which consists of over 2 hours of continuous speech. They used two acoustic models:
- A small model with 50,000 Gaussians and 2,000 mixtures.
- A large model with 150,000 Gaussians and 6,000 mixtures. Both models used 1024 clusters, with 256 active in each frame. The vocabulary contained 84,000 words and 90,000 pronunciations, and the language model included 1 million 3-grams and 300,000 4-grams.
Key Insights and Ideas
- GPU-Friendly Pruning: The paper demonstrates that hierarchical pruning, often considered difficult for GPUs, can be effectively implemented by carefully managing data layout and task distribution between CPU and GPU.
- Heterogeneous Computing is Key: The most significant speedups are achieved by intelligently assigning tasks to the most suitable processor (CPU for global optimization/pruning, GPU for massive parallel computations).
- Data Reorganization for Efficiency: Reordering Gaussian components in GPU memory to align with clusters, even if suboptimal for mixture computation, is crucial for efficient parallel evaluation of active Gaussians.
- Significant Speedup Potential: Even with the overhead of data management and synchronization, the GPU-accelerated hierarchical approach offers substantial performance gains over CPU-based methods and simpler GPU implementations.
Key Results
- The presented acoustic scorer is approximately 10x faster than its CPU implementation.
- It is approximately 2x faster than the full Gaussian evaluation performed on the GPU.
- For the large model, the acoustic scoring achieved a Real Time Factor (RTF) of about 0.05.
- This performance improvement suggests that a single GPU could theoretically support up to 20 recognizers simultaneously, leading to an approximate 30% increase in server throughput for a modest cost.
Limitations and Future Work
- Data Reorganization Overhead: While effective, the reordering of Gaussian components for cluster-based evaluation introduces some overhead. The paper notes that this is the only way to synchronize work-items in different work-groups.
- "Fake" Gaussians: To ensure equally-sized work-groups for the GPU, the authors introduced "fake" Gaussian elements, increasing the total number of Gaussians by 30%. However, the performance gains from the GPU implementation masked this redundancy.
- Future Work: The paper hints at further improvements by using a "best Gaussian approximation," which could simplify the kernel code and potentially offer additional speedups with minimal impact on accuracy. The authors also suggest that processing multiple frames together in a single kernel call is a key area for optimization.
Summary updated