Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM

Hongxu Yin, Guoyang Chen, Yingmin Li, Shuai Che, Weifeng Zhang, Niraj K. JhaarXiv 2019 · arXiv

Introduction

In the pursuit of deploying deep learning models on resource-constrained edge devices, model compression has become a standard practice. Techniques such as pruning and quantization are widely used to reduce the memory footprint and computational requirements of neural networks. However, a common assumption in the field is that "smaller is always faster." This paper challenges that notion by demonstrating that hardware-agnostic compression often fails to translate into actual execution speedups.

The authors introduce a hardware-aware training methodology for Long Short-Term Memory (LSTM) networks that accounts for the Latency Hysteresis Effect (LHE). By synchronizing model architecture with the underlying hardware characteristics, they achieve significant compression and speedups without compromising accuracy.

The Latency Hysteresis Effect (LHE)

Traditional compression strategies treat neural networks as mathematical abstractions, ignoring the physical realities of hardware execution. On modern processors (GPUs and CPUs), execution time does not decrease linearly with model size. Factors such as cache line granularity, memory alignment, and Single Instruction Multiple Data (SIMD) vectorization create a non-monotonic relationship between model dimensions and latency.

Inference latency on an NVIDIA P100 GPU showing the Latency Hysteresis Effect (LHE) and specific Latency Hysteresis Points (LHPs).

As shown in the figure above, a model with a smaller hidden layer width can actually be slower than a larger one if it falls into a "latency hysteresis bin." The authors identify Latency Hysteresis Points (LHPs)—specific dimensions where the hardware performs most efficiently. In some design spaces, over 90% of possible dimensions are sub-optimal due to these hardware quirks.

Symbiotic Training Methodology

To exploit these hardware characteristics, the authors propose a "symbiotic" training flow. This process begins with an H-LSTM (High-capacity LSTM) seed architecture, which utilizes deeper internal gates to maintain accuracy even under high sparsity. The training then proceeds through a four-stage synthesis process:

  1. Weight Growth (wg): Starting from a sparse seed, the model grows connections that are most effective at reducing the loss function.
  2. Row/Column Pruning (rcp): The model dimensions are aggressively reduced to minimize parameters and computation.
  3. Hardware-Guided Row/Column Growth (rcg): This is the critical "symbiotic" step. The model is profiled on the target hardware to find the nearest LHPs. If the pruned model is in a high-latency zone, its dimensions are grown back to the nearest hardware-friendly LHP.
  4. Weight Pruning (wp): Finally, fine-grained weight pruning is applied to maximize compactness while preserving the hardware-optimized dimensions.
The proposed four-stage symbiotic training flow: weight growth, row/column pruning, hardware-guided growth, and fine-grained weight pruning.

Experimental Evaluation

The methodology was evaluated on the Penn Treebank (PTB) dataset for language modeling and the AN4 dataset for speech recognition using the DeepSpeech2 architecture. Testing was conducted across various hardware platforms, including NVIDIA P100 and V100 GPUs and Intel Xeon CPUs.

Key Results:

  • Language Modeling (PTB): The approach reduced parameters by up to 30.5x on CPUs and 18.0x on GPUs. More importantly, it delivered inference speedups of 5.2x (CPU) and 2.4x (GPU) while maintaining baseline perplexity.
  • Speech Recognition (AN4): The model achieved 7.0x to 19.4x parameter reduction. Remarkably, the Word Error Rate (WER) improved from 12.9% to 9.9%, demonstrating that hardware-guided growth can actually enhance model accuracy by finding better architectural configurations.
  • Library Insights: The study found that specialized sparse libraries (e.g., cuSPARSE) often underperform compared to dense libraries (e.g., cuBLAS) unless sparsity exceeds 95%. This reinforces the importance of structured "dimension-based" pruning over unstructured weight removal.

Conclusion and Future Work

Hardware-Guided Symbiotic Training proves that model efficiency is a product of both algorithmic sparsity and hardware alignment. While the current work focuses on LSTMs, the principles of LHE and LHPs are likely applicable to other architectures like Transformers and CNNs. The primary trade-off is hardware specificity; because the training is guided by a specific hardware profile, the resulting model is optimized for that specific target. Future research may look into generalizing these profiles across broader hardware families to reduce the overhead of the profiling phase.

Summary updated

Abstract

Many long short-term memory (LSTM) applications need fast yet compact models. Neural network compression approaches, such as the grow-and-prune paradigm, have proved to be promising for cutting down network complexity by skipping insignificant weights. However, current compression strategies are mostly hardware-agnostic and network complexity reduction does not always translate into execution efficiency. In this work, we propose a hardware-guided symbiotic training methodology for compact, accurate, yet execution-efficient inference models. It is based on our observation that hardware may introduce substantial non-monotonic behavior, which we call the latency hysteresis effect, when evaluating network size vs. inference latency. This observation raises question about the mainstream smaller-dimension-is-better compression strategy, which often leads to a sub-optimal model architecture. By leveraging the hardware-impacted hysteresis effect and sparsity, we are able to achieve the symbiosis of model compactness and accuracy with execution efficiency, thus reducing LSTM latency while increasing its accuracy. We have evaluated our algorithms on language modeling and speech recognition applications. Relative to the traditional stacked LSTM architecture obtained for the Penn Treebank dataset, we reduce the number of parameters by 18.0x (30.5x) and measured run-time latency by up to 2.4x (5.2x) on Nvidia GPUs (Intel Xeon CPUs) without any accuracy degradation. For the DeepSpeech2 architecture obtained for the AN4 dataset, we reduce the number of parameters by 7.0x (19.4x), word error rate from 12.9% to 9.9% (10.4%), and measured run-time latency by up to 1.7x (2.4x) on Nvidia GPUs (Intel Xeon CPUs). Thus, our method yields compact, accurate, yet execution-efficient inference models.
BibTeX
@misc{yin2019hardware,
  author = {Hongxu Yin and Guoyang Chen and Yingmin Li and Shuai Che and Weifeng Zhang and Niraj K. Jha},
  title = {{Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM}},
  year = {2019},
  eprint = {1901.10997},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1901.10997},
}
Figure