Abstract
This research introduces a parallelized implementation of Deep Stacking Networks (DSNs), a deep learning architecture optimized for execution on CPU clusters. By transitioning from traditional sequential mini-batch training to a parallelized "full-batch" approach, the authors achieved substantial reductions in speech classification error rates. The study demonstrates that DSNs can effectively leverage existing CPU infrastructure to achieve state-of-the-art performance on large-scale speech datasets like TIMIT and WSJ0.
Introduction
In the early 2010s, the scalability of deep learning models was a primary concern. Popular architectures such as Deep Belief Networks (DBNs) and standard Deep Neural Networks (DNNs) relied heavily on Stochastic Gradient Descent (SGD). While effective, SGD's reliance on small, sequential mini-batches makes it difficult to parallelize across distributed systems due to the high frequency of inter-node communication required. This paper addresses the challenge of training deep models on massive datasets using standard CPU clusters, proposing a parallelization strategy for the Deep Stacking Network (DSN) architecture.
Methodology: The Deep Stacking Network
The Deep Stacking Network (DSN) is designed to be inherently more parallelizable than traditional feed-forward networks. A DSN is composed of a series of modules, or "blocks," stacked sequentially.
Architecture and Optimization
Each block consists of an input layer, a hidden layer with sigmoid nonlinearity, and a linear output layer. The "stacking" process involves concatenating the output of a previous block with the original raw input to form the input for the subsequent block. A key advantage of this structure is that the weights for the linear output layer can be solved using a closed-form mathematical solution (a least-squares problem) rather than iterative gradient-based guessing.
Parallelization Strategy
The authors parallelized the computation of the error gradient by distributing the training dataset across multiple CPUs. Each processor calculates local matrix multiplications, which are then aggregated by "accumulators" to produce a global gradient. This efficiency allows for "full-batch" training, where the entire dataset—often millions of frames—is processed in a single update step, a process typically constrained by memory on single-machine setups.
Experimental Setup
The proposed system was evaluated on two benchmark speech datasets:
- TIMIT: A standard dataset for acoustic-phonetic studies, containing approximately 1.12 million training frames.
- WSJ0 (Wall Street Journal): A larger dataset comprising 15 hours of speech, totaling roughly 5.23 million training frames.
Results and Discussion
The primary finding of this study is the significant impact of batch size on DSN performance. Unlike some architectures that experience diminishing returns with larger batches, DSNs showed a strict decrease in error rates as the batch size increased toward the full dataset.
Accuracy Gains
On the TIMIT dataset, increasing the batch size from 1,000 samples to the full-batch of 1.12 million samples resulted in the phone classification error rate dropping from 29.66% to 22.20%. Similar improvements were observed on the WSJ0 dataset, where full-batch training achieved a phone error rate of 20.99%.
Scaling and Efficiency
The parallel implementation achieved a 3x speedup when distributed across 4 to 10 processors. However, the researchers noted a classic distributed computing bottleneck: as the number of processors increased beyond 10, the overhead from inter-process communication and network disk I/O began to outweigh the computational gains, leading to increased wall-clock times.
Conclusion
This work demonstrates that carefully designed deep architectures like the DSN can achieve high performance on standard CPU clusters without the need for specialized GPU hardware. By enabling full-batch training through efficient parallelization, the authors provided a scalable alternative to traditional mini-batch methods. Future work will focus on refining the DSN architecture for more complex tasks such as full speech recognition and information retrieval, as well as optimizing communication protocols to further improve scaling.
Summary updated