Introduction
In the pursuit of deploying deep learning models on resource-constrained edge devices, model compression has become a standard practice. Techniques such as pruning and quantization are widely used to reduce the memory footprint and computational requirements of neural networks. However, a common assumption in the field is that "smaller is always faster." This paper challenges that notion by demonstrating that hardware-agnostic compression often fails to translate into actual execution speedups.
The authors introduce a hardware-aware training methodology for Long Short-Term Memory (LSTM) networks that accounts for the Latency Hysteresis Effect (LHE). By synchronizing model architecture with the underlying hardware characteristics, they achieve significant compression and speedups without compromising accuracy.
The Latency Hysteresis Effect (LHE)
Traditional compression strategies treat neural networks as mathematical abstractions, ignoring the physical realities of hardware execution. On modern processors (GPUs and CPUs), execution time does not decrease linearly with model size. Factors such as cache line granularity, memory alignment, and Single Instruction Multiple Data (SIMD) vectorization create a non-monotonic relationship between model dimensions and latency.
As shown in the figure above, a model with a smaller hidden layer width can actually be slower than a larger one if it falls into a "latency hysteresis bin." The authors identify Latency Hysteresis Points (LHPs)—specific dimensions where the hardware performs most efficiently. In some design spaces, over 90% of possible dimensions are sub-optimal due to these hardware quirks.
Symbiotic Training Methodology
To exploit these hardware characteristics, the authors propose a "symbiotic" training flow. This process begins with an H-LSTM (High-capacity LSTM) seed architecture, which utilizes deeper internal gates to maintain accuracy even under high sparsity. The training then proceeds through a four-stage synthesis process:
- Weight Growth (wg): Starting from a sparse seed, the model grows connections that are most effective at reducing the loss function.
- Row/Column Pruning (rcp): The model dimensions are aggressively reduced to minimize parameters and computation.
- Hardware-Guided Row/Column Growth (rcg): This is the critical "symbiotic" step. The model is profiled on the target hardware to find the nearest LHPs. If the pruned model is in a high-latency zone, its dimensions are grown back to the nearest hardware-friendly LHP.
- Weight Pruning (wp): Finally, fine-grained weight pruning is applied to maximize compactness while preserving the hardware-optimized dimensions.
Experimental Evaluation
The methodology was evaluated on the Penn Treebank (PTB) dataset for language modeling and the AN4 dataset for speech recognition using the DeepSpeech2 architecture. Testing was conducted across various hardware platforms, including NVIDIA P100 and V100 GPUs and Intel Xeon CPUs.
Key Results:
- Language Modeling (PTB): The approach reduced parameters by up to 30.5x on CPUs and 18.0x on GPUs. More importantly, it delivered inference speedups of 5.2x (CPU) and 2.4x (GPU) while maintaining baseline perplexity.
- Speech Recognition (AN4): The model achieved 7.0x to 19.4x parameter reduction. Remarkably, the Word Error Rate (WER) improved from 12.9% to 9.9%, demonstrating that hardware-guided growth can actually enhance model accuracy by finding better architectural configurations.
- Library Insights: The study found that specialized sparse libraries (e.g., cuSPARSE) often underperform compared to dense libraries (e.g., cuBLAS) unless sparsity exceeds 95%. This reinforces the importance of structured "dimension-based" pruning over unstructured weight removal.
Conclusion and Future Work
Hardware-Guided Symbiotic Training proves that model efficiency is a product of both algorithmic sparsity and hardware alignment. While the current work focuses on LSTMs, the principles of LHE and LHPs are likely applicable to other architectures like Transformers and CNNs. The primary trade-off is hardware specificity; because the training is guided by a specific hardware profile, the resulting model is optimized for that specific target. Future research may look into generalizing these profiles across broader hardware families to reduce the overhead of the profiling phase.
Summary updated