Introduction
Modern Deep Neural Networks (DNNs) have achieved state-of-the-art performance across various domains but at the cost of extreme over-parameterization. These models often contain millions or billions of parameters, leading to significant spatial complexity (storage/memory) and temporal complexity (training time). This paper provides a comprehensive survey of low-rank optimization techniques designed to exploit inherent redundancy in DNNs, bridging the gap between model compression and accelerated training.
A Unified Framework for Efficiency
The core contribution of this work is a conceptual framework that categorizes low-rank optimization into three primary training paradigms:
- Pre-train Methods: These involve decomposing a pre-trained large model and fine-tuning it to recover performance.
- Pre-set Methods: These define compact architectures (e.g., Tensor Train or Tensor Ring) from the outset and train them from scratch.
- Compression-aware Methods: These use specialized regularizers during the training of a standard model to induce a low-rank structure.
Beyond architecture, the paper highlights Subspace Training as a means to achieve temporal efficiency. By projecting parameters into a low-dimensional subspace—often identified via PCA on training trajectories—researchers can employ faster-converging second-order optimization methods that would otherwise be computationally prohibitive in the full parameter space.
The Spatial-Temporal Balance
A key insight of the paper is the intrinsic link between spatial and temporal redundancy. The authors argue that redundancy provides the "wiggle room" necessary for efficient optimization. Aggressive compression can remove this flexibility, potentially slowing down training. Finding the "sweet spot" is crucial for practical deployment.
Furthermore, the authors advocate for Effective Rank (based on the Shannon entropy of singular values) over the traditional Nuclear Norm. Their findings suggest that Effective Rank is a superior metric for driving compression without sacrificing the model's expressive power, achieving significantly higher compression ratios at comparable accuracy levels.
Key Results and Performance
The survey evaluates various tensor decomposition methods across standard benchmarks. Notably, advanced techniques such as Semi-tensor Product-based (STT) methods demonstrate the ability to achieve high compression ratios while maintaining competitive accuracy.
As shown in the table above, STT achieves a 9x compression ratio with minimal accuracy loss. Additionally, subspace training methods like DLDR have been shown to reduce training time by approximately 35% compared to standard Stochastic Gradient Descent (SGD) by optimizing in a reduced-dimension space.
Future Directions
While low-rank optimization shows great promise, several challenges remain:
- Hardware Realization: Reducing FLOPs does not always lead to lower latency due to memory access patterns and lack of specialized kernel support for non-standard tensor formats.
- Energy Efficiency: Future research must move beyond compute metrics to evaluate the total energy footprint of these models.
- End-to-End Subspace Optimization: Developing methods that operate entirely within the low-rank subspace, avoiding high-dimensional gradient calculations, remains a major goal.
Summary updated