Low-rank Embedding of Kernels in Convolutional Neural Networks under Random Shuffling

Chao Li, Zhun Sun, Jinshi Yu, Ming Hou, Qibin Zhao

TL;DR

This paper challenges the conventional wisdom that Convolutional Neural Network (CNN) kernels are compressible only because of spatial patterns in images. By showing that kernels can be effectively compressed even after their weights are randomly scrambled, the authors demonstrate that "low-rankness" is an inherent property of neural network learning, rather than just a reflection of spatial similarity in training data.

What Problem Does This Paper Address?

Deep learning models like AlexNet and VGG are notoriously "heavy," often exceeding hundreds of megabytes in size. This makes deploying them on resource-constrained devices (like smartphones or IoT sensors) difficult.

A popular solution is Tensor Decomposition (TD), which treats the weights of a convolutional layer as a multi-dimensional array (a tensor) and breaks it down into smaller, low-rank components. Historically, researchers believed TD worked because images have high spatial similarity—pixels near each other are often related—and the network learns kernels that reflect this linear dependence. The authors of this paper ask a fundamental question: Is this spatial similarity actually the essential cause of the low-rank structure in kernels?

Methods

The authors introduce a framework called Randomly-shuffled Tensor Decomposition (RsTD).

  1. The Shuffle: Before applying decomposition, they use a linear operator ($R$) to randomly re-assign the indices of every parameter in the kernel. This effectively "scrambles" any spatial or channel-wise patterns that were learned during training.
  2. Unified Tensor Model: They use a unified mathematical framework to represent various TD methods, including Tensor Train (TT), TT-matrix, and Tensor Ring (TR).
  3. Training: Instead of decomposing a pre-trained model, they directly update the "latent core tensors" (the compressed pieces) during the training process using standard backpropagation.

By comparing standard TD (which preserves spatial order) with RsTD (which destroys it), they can isolate whether spatial structure is necessary for high-ratio compression.

Datasets

The study evaluates its approach using:

  • CIFAR-10: A standard benchmark for image classification.
  • Noised CIFAR-10: The authors added Additive White Gaussian Noise (AWGN) with varying standard deviations (0.4 and 0.8) to the images. This was done to intentionally decrease the spatial similarity of the input data and see how it affected the kernels' compressibility.

Key Insights and Ideas

  • Inherent Low-Rank Structure: The most surprising finding is that RsTD performs almost as well as—and sometimes better than—standard TD. This implies that the ability to represent a kernel in a low-rank subspace is a fundamental characteristic of how CNNs learn, independent of the physical arrangement of the weights.
  • Mitigating Under-fitting: At extreme compression ratios (e.g., compressing a model to 1% of its original size), standard TD often fails. The authors explain that in standard TD, a very low rank (like rank-1) forces all output channels to be identical versions of each other, just scaled differently.
  • Shuffling as a Feature Booster: Random shuffling (RsTD) actually helps at high compression levels. Because the weights are scrambled, the "latent features" created during the convolution are more diverse. This prevents the "under-fitting" seen in standard TD and leads to more stable classification accuracy at tiny parameter counts.

Key Results

  • 20x Compression: Both TD and RsTD achieved significant compression (up to a ratio of 0.05) on CIFAR-10 with negligible loss in accuracy compared to the uncompressed baseline.
  • Superiority at the Edge: At very high compression ratios (below 0.01), RsTD significantly outperformed standard TD. For example, in the TT-matrix experiments, RsTD maintained much higher accuracy while standard TD's performance plummeted.
  • Noise Resilience: When training on noised data (where spatial similarity is lower), the performance gap between standard TD and RsTD narrowed. This confirms that while spatial similarity helps standard TD, it isn't the primary reason why these networks can be compressed.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Scale: The experiments were conducted on CIFAR-10 with a relatively small 8-layer CNN. It remains to be seen if these findings hold for massive architectures like ResNet-152 or DenseNet on ImageNet.
  • Computational Overhead: While the paper focuses on storage (parameter count), the random shuffling operator adds a layer of complexity to the actual convolution operation that might impact inference speed on certain hardware.
  • Theoretical Proof: The paper provides strong empirical evidence for "inherent low-rankness," but a formal theoretical proof of why neural networks converge to these structures regardless of weight ordering is still an open challenge.

Summary updated

Abstract

Although the convolutional neural networks (CNNs) have become popular for various image processing and computer vision tasks recently, it remains a challenging problem to reduce the storage cost of the parameters for resource-limited platforms. In the previous studies, tensor decomposition (TD) has achieved promising compression performance by embedding the kernel of a convolutional layer into a low-rank subspace. However the employment of TD is naively on the kernel or its specified variants. Unlike the conventional approaches, this paper shows that the kernel can be embedded into more general or even random low-rank subspaces. We demonstrate this by compressing the convolutional layers via randomly-shuffled tensor decomposition (RsTD) for a standard classification task using CIFAR-10. In addition, we analyze how the spatial similarity of the training data influences the low-rank structure of the kernels. The experimental results show that the CNN can be significantly compressed even if the kernels are randomly shuffled. Furthermore, the RsTD-based method yields more stable classification accuracy than the conventional TD-based methods in a large range of compression ratios.
BibTeX
@inproceedings{li2019low,
  author = {Chao Li and Zhun Sun and Jinshi Yu and Ming Hou and Qibin Zhao},
  title = {{Low-rank Embedding of Kernels in Convolutional Neural Networks under Random Shuffling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=GI6FBjiktQE},
}

Loading related content...

Figure