Abstract
Deep neural networks (DNNs) face significant deployment challenges on resource-constrained edge devices due to their substantial storage and computational requirements. This paper introduces ADA-Tucker, a novel compression framework that adaptively reshapes weight tensors into balanced, higher-order dimensions before applying Tucker decomposition. By optimizing the tensor's shape and order, ADA-Tucker achieves more compact low-rank approximations than standard decomposition methods. Additionally, the authors propose SCADA-Tucker (Shared Core ADA-Tucker), which leverages cross-layer redundancy by sharing a single core tensor across all network layers. Experimental results demonstrate that these techniques, combined with quantization and Huffman coding, can achieve massive compression ratios—up to 691× on LeNet-5—with minimal impact on accuracy.
Introduction
The increasing complexity of deep learning models has led to a surge in storage and memory demands, making them difficult to deploy on mobile and edge hardware. While traditional compression methods like pruning, quantization, and standard low-rank decompositions have been effective, they often suffer from limitations inherent in the original tensor shapes.
Existing low-rank models typically decompose weight tensors in their native formats (e.g., 4D for convolutional layers, 2D for fully-connected layers). When these dimensions are imbalanced—such as a $1 \times 100$ matrix—the achievable compression is constrained by the smallest dimension, leading to significant information loss during rank reduction. ADA-Tucker addresses this by reshaping tensors into more "balanced" configurations to facilitate better low-rank approximation.
Methodology
Adaptive Dimension Adjustment (ADA)
The core innovation of ADA-Tucker is the adaptive adjustment of the weight tensor's shape prior to decomposition. Instead of treating a convolutional kernel as a fixed 4D tensor, the framework reshapes it into a higher-order tensor (e.g., 5D or 6D) or merges dimensions to create a more balanced, "square" shape.
As illustrated in Figure 3, balancing the dimensions of each mode allows the Tucker decomposition to find a much more compact representation. A balanced tensor is significantly easier to approximate with low-rank methods than an imbalanced one, preserving the expressive power of the weights at higher compression ratios.
Shared Core ADA-Tucker (SCADA-Tucker)
Building on the ADA concept, the authors identify that different layers within a network often share redundant information. SCADA-Tucker proposes that all layers in a network can share a single common core tensor $\mathcal{C}$, while maintaining unique transformation matrices for each layer.
This approach, shown in Figure 2, is conceptually similar to weight reuse in Recurrent Neural Networks (RNNs). The shared core tensor acts as a global feature extractor, while the layer-specific transformation matrices function as adapters. This significantly reduces the total number of parameters required to represent the entire network.
Experimental Results
The researchers evaluated ADA-Tucker across several standard computer vision benchmarks, including MNIST (LeNet-5, LeNet-300-100), CIFAR-10, SVHN (ResNet-20, Wide-ResNet-28-10), and ImageNet (AlexNet).
Key Findings
- Massive Compression: On LeNet-5 (MNIST), the framework achieved a 691× compression ratio with only a 0.14% drop in accuracy (from 99.2% to 99.06%).
- High-Order Benefits: Reshaping fully-connected layers into 5th-order tensors yielded better results than traditional 2D representations, allowing for more granular decomposition.
- Robustness to Decay: On the ImageNet benchmark (AlexNet), ADA-Tucker exhibited a much slower accuracy decay compared to CP-decomposition. At an 18× compression ratio, ADA-Tucker lost less than 4% accuracy, whereas CP-decomposition accuracy dropped by nearly 70%.
- Zero Metadata Overhead: Unlike pruning, which requires storing indices for non-zero elements, ADA-Tucker produces dense, structured tensors that do not require additional bookkeeping.
Conclusion
ADA-Tucker provides a flexible and powerful framework for DNN compression by rethinking the "natural" shapes of weight tensors. By balancing dimensions and sharing core information across layers, it pushes the boundaries of achievable compression ratios without sacrificing performance. Future work will focus on automating the selection of optimal tensor orders and exploring the synergy between ADA-Tucker and advanced pruning techniques.
Summary updated