Abstract
Although convolutional neural networks (CNNs) have become popular for various image processing and computer vision tasks recently, it remains a challenging problem to reduce the storage cost of parameters for resource-limited platforms. In previous studies, tensor decomposition (TD) has achieved promising compression performance by embedding the kernel of a convolutional layer into a low-rank subspace. However, the employment of TD is naively on the kernel or its specified variants. Unlike conventional approaches, this paper shows that the kernel can be embedded into more general or even random low-rank subspaces. We demonstrate this by compressing convolutional layers via randomly-shuffled tensor decomposition (RsTD) for a standard classification task using CIFAR-10. In addition, we analyze how the spatial similarity of training data influences the low-rank structure of the kernels. The experimental results show that the CNN can be significantly compressed even if the kernels are randomly shuffled. Furthermore, the RsTD-based method yields more stable classification accuracy than conventional TD-based methods in a large range of compression ratios.
Introduction
The deployment of state-of-the-art deep learning models like VGG or ResNet on edge devices is often hindered by their massive parameter counts. Tensor Decomposition (TD) has emerged as a powerful technique to mitigate this, treating convolutional kernels as multi-dimensional arrays and decomposing them into smaller, low-rank components.
A prevailing assumption in the field has been that TD's effectiveness stems from the high spatial similarity inherent in natural images. It was believed that networks learn kernels with linear dependencies reflecting this spatial structure. This paper challenges that assumption by asking: Is spatial similarity truly the essential cause of low-rank structure in kernels?
Methodology: Randomly-shuffled Tensor Decomposition (RsTD)
To investigate the necessity of spatial structure, the authors propose Randomly-shuffled Tensor Decomposition (RsTD). The core idea is to apply a random linear permutation to the kernel indices before decomposition, effectively destroying any learned spatial or channel-wise patterns.
The framework utilizes a unified mathematical model to represent various decomposition formats, such as Tensor Train (TT) and Tensor Ring (TR), as shown in the figure above. Instead of decomposing a pre-trained model, the "latent core tensors" are trained directly from scratch using standard backpropagation. This allows the network to adapt its weights to a low-rank subspace that is not constrained by the original physical arrangement of parameters.
Experimental Results and Insights
The approach was evaluated on the CIFAR-10 dataset and a "Noised CIFAR-10" variant (using Additive White Gaussian Noise) to specifically reduce input spatial similarity.
Inherent Low-Rankness
The most significant finding is that RsTD performs comparably to, and often exceeds, standard TD. As seen in the results, both methods achieve roughly 20x compression (ratio of 0.05) with minimal accuracy loss. This suggests that low-rankness is an inherent property of neural network optimization rather than a mere byproduct of spatial data patterns.
Stability at Extreme Compression
At very high compression ratios (below 0.01), standard TD performance often collapses. In standard TD, extreme rank reduction forces output channels to become nearly identical, leading to severe under-fitting. Conversely, RsTD maintains much higher stability. By scrambling the weights, the latent features generated during convolution remain more diverse, preventing the "feature collapse" seen in ordered decompositions.
Resilience to Noise
When training on noised data, the performance gap between standard TD and RsTD narrows. This confirms that while spatial similarity can assist standard TD, it is not the primary driver of compressibility. The network's ability to find a low-rank representation persists even when the input data's spatial structure is degraded.
Conclusion and Future Work
This research demonstrates that CNN kernels possess an inherent low-rank structure that is independent of weight ordering. By utilizing random shuffling, models can be compressed more stably at extreme ratios than with conventional methods.
While these results are promising on CIFAR-10, future work should explore the scalability of RsTD on larger architectures and datasets like ImageNet. Additionally, the computational overhead of the shuffling operator during inference warrants further investigation to ensure that storage savings translate into practical performance gains on hardware.
Summary updated