Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training

Xinwei Ou, Zhangxin Chen, Ce Zhu, Yipeng LiuarXiv 2023 · DOI · arXiv

Introduction

Modern Deep Neural Networks (DNNs) have achieved state-of-the-art performance across various domains but at the cost of extreme over-parameterization. These models often contain millions or billions of parameters, leading to significant spatial complexity (storage/memory) and temporal complexity (training time). This paper provides a comprehensive survey of low-rank optimization techniques designed to exploit inherent redundancy in DNNs, bridging the gap between model compression and accelerated training.

A Unified Framework for Efficiency

The core contribution of this work is a conceptual framework that categorizes low-rank optimization into three primary training paradigms:

  1. Pre-train Methods: These involve decomposing a pre-trained large model and fine-tuning it to recover performance.
  2. Pre-set Methods: These define compact architectures (e.g., Tensor Train or Tensor Ring) from the outset and train them from scratch.
  3. Compression-aware Methods: These use specialized regularizers during the training of a standard model to induce a low-rank structure.
A unified framework for low-rank optimization, balancing spatial and temporal efficiency.

Beyond architecture, the paper highlights Subspace Training as a means to achieve temporal efficiency. By projecting parameters into a low-dimensional subspace—often identified via PCA on training trajectories—researchers can employ faster-converging second-order optimization methods that would otherwise be computationally prohibitive in the full parameter space.

The Spatial-Temporal Balance

A key insight of the paper is the intrinsic link between spatial and temporal redundancy. The authors argue that redundancy provides the "wiggle room" necessary for efficient optimization. Aggressive compression can remove this flexibility, potentially slowing down training. Finding the "sweet spot" is crucial for practical deployment.

Furthermore, the authors advocate for Effective Rank (based on the Shannon entropy of singular values) over the traditional Nuclear Norm. Their findings suggest that Effective Rank is a superior metric for driving compression without sacrificing the model's expressive power, achieving significantly higher compression ratios at comparable accuracy levels.

Key Results and Performance

The survey evaluates various tensor decomposition methods across standard benchmarks. Notably, advanced techniques such as Semi-tensor Product-based (STT) methods demonstrate the ability to achieve high compression ratios while maintaining competitive accuracy.

Performance comparison of various low-rank tensor decomposition methods.

As shown in the table above, STT achieves a 9x compression ratio with minimal accuracy loss. Additionally, subspace training methods like DLDR have been shown to reduce training time by approximately 35% compared to standard Stochastic Gradient Descent (SGD) by optimizing in a reduced-dimension space.

Future Directions

While low-rank optimization shows great promise, several challenges remain:

  • Hardware Realization: Reducing FLOPs does not always lead to lower latency due to memory access patterns and lack of specialized kernel support for non-standard tensor formats.
  • Energy Efficiency: Future research must move beyond compute metrics to evaluate the total energy footprint of these models.
  • End-to-End Subspace Optimization: Developing methods that operate entirely within the low-rank subspace, avoiding high-dimensional gradient calculations, remains a major goal.

Summary updated

Abstract

Deep neural networks have achieved great success in many data processing applications. However, the high computational complexity and storage cost makes deep learning hard to be used on resource-constrained devices, and it is not environmental-friendly with much power cost. In this paper, we focus on low-rank optimization for efficient deep learning techniques. In the space domain, deep neural networks are compressed by low rank approximation of the network parameters, which directly reduces the storage requirement with a smaller number of network parameters. In the time domain, the network parameters can be trained in a few subspaces, which enables efficient training for fast convergence. The model compression in the spatial domain is summarized into three categories as pre-train, pre-set, and compression-aware methods, respectively. With a series of integrable techniques discussed, such as sparse pruning, quantization, and entropy coding, we can ensemble them in an integration framework with lower computational complexity and storage. Besides of summary of recent technical advances, we have two findings for motivating future works: one is that the effective rank outperforms other sparse measures for network compression. The other is a spatial and temporal balance for tensorized neural networks.
BibTeX
@misc{ou2023low,
  author = {Xinwei Ou and Zhangxin Chen and Ce Zhu and Yipeng Liu},
  title = {{Low Rank Optimization for Efficient Deep Learning: Making A Balance between Compact Architecture and Fast Training}},
  year = {2023},
  eprint = {2303.13635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.13635},
}
Figure