Parallel training of neural networks for speech recognition

Karel Veselý, Lukáš Burget, František GrézlInterspeech 2010

TL;DR

This paper tackles the computationally intensive problem of training neural networks for speech recognition by introducing two parallelization strategies: data parallelism using POSIX threads for multi-core CPUs and node parallelism leveraging GPU acceleration with CUDA. They demonstrate significant speedups, with CUDA achieving nearly 10x faster training compared to a baseline BLAS-optimized implementation.

What Problem Does This Paper Address?

Training large neural networks for speech recognition is a time-consuming process, often taking over 24 hours even with existing optimizations. This lengthy training period hinders practical use and slows down research progress. The authors aim to significantly reduce this training time by exploring and implementing parallel processing techniques to accelerate the backpropagation algorithm used for training feed-forward Multi-Layer Perceptron (MLP) networks, which are a cornerstone of modern hybrid HMM-ANN speech recognition systems.

Methods

The paper presents two primary methods for parallelizing the training of Artificial Neural Networks (ANNs) for speech recognition:

  • Data Parallelization (POSIX Threads): This approach divides the training data into smaller, disjoint sets. Each thread then works on its own subset of data using its own instance of the network. Crucially, weight synchronization occurs periodically, where the weight differences calculated by each thread are gathered, summed, and used to update the network weights. This method is well-suited for multi-core CPU architectures.
  • Node Parallelization (GPU with CUDA): This method exploits the Single Instruction, Multiple Data (SIMD) architecture of Graphics Processing Units (GPUs) using NVIDIA's CUDA platform. Instead of distributing data, the network's layers are divided into disjoint sets of neurons, with each thread handling a specific set. This approach requires more frequent synchronization, typically at the end of each layer's computation, to ensure correct forward and backward passes. The authors leverage CUBLAS for accelerating linear algebra operations and implement custom CUDA kernels for non-linear transformations and bias updates.

Both approaches are compared against a baseline that uses the GotoBLAS library for optimized linear algebra operations.

Datasets

The primary dataset used for training and evaluation is a subset of the AMIDA meeting data corpus.

  • Training set: 135 hours of speech data.
  • Cross-validation set: 15 hours of speech data. The corpus is labeled with 45 phonemes, each modeled with three sub-states, resulting in 135 output classes for the network.

Key Insights and Ideas

  • GPU Acceleration Dominance: The results strongly suggest that for this specific task, GPU-based node parallelization (CUDA) offers a significantly greater speedup than multi-threaded data parallelization on CPUs. This highlights the power of specialized hardware for highly parallelizable computations like those in neural network training.
  • Data Transfer Bottleneck in CUDA: While CUDA offers impressive speedups, the authors identify data transfers between the host (CPU) and the GPU memory as a potential bottleneck. Minimizing these transfers by processing data in larger segments is crucial for maximizing efficiency.
  • Trade-offs in Parallelization: Data parallelization scales with the number of threads but faces overheads in weight synchronization. Node parallelization offers higher parallelism within a single node but requires more frequent synchronization.
  • Importance of BLAS Optimization: Even the baseline single-threaded implementation benefits significantly from optimized linear algebra libraries like GotoBLAS, underscoring the foundational role of efficient matrix operations in neural network training.

Key Results

  • CUDA Speedup: The CUDA version of the training achieved a speedup of 9.7x compared to the BLAS-optimized single-thread baseline. This significantly outperformed the 8-core multi-thread version.
  • Multi-thread Speedup: The multi-thread data parallelization on an 8-core server achieved a speedup of 4.4x compared to the baseline.
  • CUDA Outperforms Multi-thread: The CUDA version was approximately 2.2x faster than the 8-core multi-thread version.
  • Profiling Insights: Profiling revealed that for CUDA, data preprocessing and CPU-GPU transfers accounted for a substantial portion of the running time (37% and 8% respectively), while for the multi-thread version, the training itself (87%) and waiting on barriers were the main contributors.

Limitations and Future Work

  • Limitations:
    • The paper notes that data parallelization doesn't scale ideally and synchronization overhead increases with more threads.
  • For CUDA, data transfers between host and GPU can be a bottleneck.
  • The scaling of the multi-thread version on a 32-core server showed performance degradation beyond a certain number of cores, identified as a RAM to CPU bandwidth bottleneck.
  • Future Work:
    • The authors plan to release the TNet tool as open-source software.
  • Future work could focus on scaling by using multiple GPUs within a single PC.
  • Exploring the use of a second GPU to accelerate feature preprocessing is also suggested for further speed improvements.
  • Further investigation into optimal aggregated bunch-sizes for weight updates is mentioned.

Summary updated

BibTeX
@inproceedings{vesely2010parallel,
  author = {Karel Veselý and Lukáš Burget and František Grézl},
  title = {{Parallel training of neural networks for speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure