Abstract: Advanced tensor decomposition, such as tensor train (TT), has been widely studied for tensor decomposition-based neural network (NN) training, which is one of the most common model compression methods. However, training NN with tensor decomposition always suffers significant accuracy loss and convergence issues. In this paper, a holistic framework is proposed for tensor decomposition-based NN training by formulating TT decomposition-based NN training as a nonconvex optimization problem. This problem can be solved by the proposed tensor block coordinate descent (tenBCD) method, which is a gradient-free algorithm. The global convergence of tenBCD to a critical point at a rate of O(1/k) is established with the Kurdyka-Łojasiewicz (KŁ) property, where k is the number of iterations. The theoretical results can be extended to the popular residual neural networks (ResNets). The effectiveness and efficiency of our proposed framework are verified through an image classification dataset, where our proposed method can converge efficiently in training and prevent overfitting.
The Challenge of Model Compression
As neural networks grow in size—reaching hundreds of billions of parameters—deploying them on edge devices or IoT systems becomes nearly impossible due to memory and power constraints. Researchers typically employ low-rank approximation (reducing matrix complexity) or weight pruning (removing redundant connections) to shrink these models.
However, these techniques often incur a "training tax." Training a compressed model from scratch frequently leads to poor accuracy because the reduced capacity makes the model hypersensitive to initialization. Conversely, pruning a pre-trained model often introduces significant approximation errors that are difficult to recover via standard fine-tuning. Furthermore, both approaches typically rely on gradient-based optimization (like SGD), which suffers from vanishing or exploding gradients when dealing with the complex, non-linear structures of compressed networks.
The NN-BCD Framework
To address these challenges, this paper introduces NN-BCD (Neural Network Block Coordinate Descent), a gradient-free training framework that treats model compression as a unified nonconvex optimization problem.
The framework is built on several key innovations:
- Variable Splitting: The authors introduce auxiliary variables for each layer's inputs, outputs, and weights. This "decouples" the layers, turning a nested, non-linear system into a series of manageable subproblems.
- Gradient-Free Updates: Instead of backpropagating errors, NN-BCD updates one "block" of variables at a time while keeping others fixed. These updates are derived in closed-form (using least squares or proximal operators). Because it avoids gradients, the algorithm is immune to the vanishing/exploding gradient problem.
- Unified Compression: The framework utilizes a generalized "model compression operator." By adjusting this operator, the same mathematical foundation can perform either Tensor Train Decomposition (TTD) or magnitude-based weight pruning.
Theoretical and Empirical Results
Using the Kurdyka-Łojasiewicz (KŁ) property, the authors prove that their iterative sequence is guaranteed to converge to a critical point at a rate of $O(1/k)$. This theoretical stability was verified across several distinct tasks:
- Extreme Compression on MNIST: The team achieved a Compression Ratio (CR) of 0.0128 (nearly 80x smaller) while maintaining a test accuracy of 97.38%—actually outperforming the uncompressed baseline (97.09%).
- High Sparsity on Flare Data: On a highly imbalanced space-weather dataset, they reached a sparsity level of 0.9977 (compressing the model over 434 times) while maintaining a Balanced Accuracy (BACC) of 0.8890.
- Superiority over SGD: Compared to standard gradient-based optimizers (Adam, RMSprop), NN-BCD showed much smoother convergence and higher final accuracy, regardless of the weight initialization method used.
Insights and Future Directions
A striking finding of this work is that compression can act as a powerful regularizer. Instead of simply "chopping" a model, the objective function includes a penalty term that encourages weights to stay close to a compressed version, allowing the model to learn a compressed representation gracefully.
While the current NN-BCD algorithm is designed for batch training, future work will explore a stochastic version for massive datasets and the integration of quantization (reducing bit-precision) into the unified framework. Despite the memory overhead of auxiliary variables during training, the resulting models are significantly more efficient for deployment.
Summary updated