Introduction
Deep neural networks (DNNs) have achieved state-of-the-art performance across various domains but are often too memory-intensive for deployment on resource-constrained edge devices. Tensor decomposition offers a promising solution by breaking down large weight matrices into smaller, interconnected "cores" using formats like Tensor Train (TT) or Tensor Ring (TR). However, the effectiveness of these methods hinges on the selection of the decomposition rank, a hyperparameter that balances accuracy and compression. Finding optimal ranks for every layer is a discrete, combinatorial search problem that typically requires expensive iterative retraining or complex genetic algorithms.
This paper introduces a novel end-to-end framework that automates rank selection by treating it as a sparsity problem. By employing a probabilistic ℓ0-regularization approach with "Bernoulli gates," the proposed method enables a model to learn its weights and its compression architecture simultaneously via standard gradient descent.
Methodology: Probabilistic ℓ0-Regularized Rank Selection
The authors propose a differentiable mechanism to prune tensor ranks during training. The approach consists of several key innovations:
Gated Tensor Contraction
The core idea is the insertion of diagonal matrices, termed Bernoulli gates ($Z_i$), between the tensor cores. Each entry in these matrices acts as a binary "switch" for a specific slice of the tensor core. When a gate entry becomes zero, the corresponding slice is effectively pruned, thereby reducing the operational rank of the decomposition.
Continuous Relaxation and ℓ0 Penalty
To make the discrete rank selection differentiable, the authors utilize a Gaussian-based Bernoulli relaxation. This turns the hard on/off switches into probabilistic variables that can be optimized using backpropagation. To encourage sparsity (and thus higher compression), an ℓ0 penalty term is added to the loss function, directly penalizing the number of active ranks.
Variance-Preserving Initialization
Training tensorized networks from scratch can be unstable. The authors derived a specific initialization scheme for Tensor Ring (TR) cores, analogous to Kaiming initialization, to ensure stable signal propagation during the initial training phases.
Experimental Evaluation and Results
The method was evaluated on diverse tasks, including image classification (MNIST, CIFAR-10), text sentiment analysis (SST-5), and neural machine translation (IWSLT’14 German-to-English).
Extreme Compression and Accuracy
On a 2-layer MLP for MNIST, the method achieved a remarkable 240x compression ratio while maintaining 97.5% accuracy, outperforming previous state-of-the-art methods like MARS and LR-BTNN. Similar gains were observed for LeNet-5, where the method reached 61.39x compression with 99.19% accuracy, significantly higher than the 26x compression achieved by uniform-rank Tensor Ring approaches.
Scalability to Complex Architectures
In Transformer models, the approach reduced the embedding layer size by over 30x without significant degradation in BLEU scores. For ResNet-32 on CIFAR-10, the method maintained over 90% accuracy while achieving 5x compression, demonstrating its applicability to deep, modern architectures.
Key Insights and Discussion
Simultaneous Learning vs. Search
Unlike traditional methods that compress pre-trained models, this approach learns the "compressed shape" and the weights concurrently. This allows the weights to adapt to the evolving rank structure during training. Furthermore, it is highly efficient; while genetic algorithm-based searches can take days, this method finds optimal ranks in roughly the same time required to train a standard tensorized network (~0.11 GPU days).
The Importance of Non-Uniformity
The results demonstrate that the model naturally converges to non-uniform rank distributions across layers. Some layers exhibit high redundancy and can be compressed aggressively, while others require higher capacity. This nuance is captured automatically, avoiding the sub-optimality of uniform-rank heuristics.
Limitations and Future Directions
While effective, the probabilistic gates can introduce stochasticity that may destabilize very deep networks like ResNet. The authors addressed this with a "budget-aware" scheme to stabilize gates once target compression is reached. Future work may explore combining this rank selection with quantization or knowledge distillation to push the boundaries of model efficiency even further.
Summary updated