Reduced storage direct tensor ring decomposition for convolutional neural networks compression

Mateusz Gabor, Rafal ZdunekNeural Networks 2026 · DOI

TL;DR

This paper introduces a novel method called Reduced Storage Direct Tensor Ring Decomposition (RSDTR) for compressing Convolutional Neural Networks (CNNs). RSDTR significantly reduces the storage and computational requirements of CNNs by efficiently decomposing their weight tensors, achieving competitive accuracy while offering substantial compression ratios.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) are incredibly powerful for tasks like image classification, but their effectiveness comes at a cost: they are often very large and computationally intensive. This makes deploying them on resource-constrained devices like smartphones or edge devices challenging due to limited storage and processing power. While various compression techniques exist (pruning, quantization, low-rank approximations), existing low-rank methods, particularly those using tensor ring (TR) decomposition, often struggle to balance parameter reduction with computational cost and accuracy. Some methods achieve high parameter compression but increase FLOPS (floating-point operations per second), or require training from scratch, which is time-consuming. This paper aims to address this by proposing a more efficient low-rank compression method that optimizes for both storage and computational efficiency without sacrificing accuracy.

Methods

The core of this paper is the Reduced Storage Direct Tensor Ring Decomposition (RSDTR) method. It builds upon the Tensor Ring (TR) decomposition, a technique for representing multi-dimensional arrays (tensors) as a chain of smaller tensors. The key innovation of RSDTR lies in its ability to explore different "circular permutations" of the TR decomposition. TR decomposition is invariant to the order of its modes, meaning the same tensor can be represented in multiple ways by cyclically shifting its modes. RSDTR systematically searches for the circular permutation that yields the most compact representation (lowest storage cost) for a given compression accuracy.

Instead of directly decomposing the large convolutional kernel tensor, RSDTR decomposes it into a series of smaller core tensors. These decomposed factors are then used as new weights, effectively replacing a single large convolutional layer with a pipeline of smaller tensor operations (contractions and 1D convolutions). This approach allows for fine-tuning the compressed network from these decomposed factors, rather than retraining from scratch. The method leverages an exhaustive search to find the optimal circular permutation and ranks, ensuring the best trade-off between compression and accuracy.

Datasets

The proposed method was evaluated on two widely used image classification datasets:

  • CIFAR-10: A dataset of 60,000 32x32 color images in 10 classes.
  • ImageNet: A large-scale dataset with millions of images across thousands of classes.

Key Insights and Ideas

  • Circular Permutation Matters for Storage: The paper highlights that while TR decomposition is theoretically invariant to mode order, the choice of circular permutation significantly impacts the resulting storage complexity. RSDTR leverages this by actively searching for the optimal permutation.
  • Direct Decomposition for Fine-tuning: Unlike some tensorization methods that require retraining from scratch, RSDTR's direct decomposition of weights allows for fine-tuning of the compressed network, which is more efficient.
  • Balanced Compression: RSDTR aims for a simultaneous reduction in both parameter count and FLOPS, a crucial advantage over methods that might reduce parameters at the expense of increased computation.
  • Efficiency on Diverse Architectures: The method was tested on various CNN architectures, from smaller ResNets (20, 32, 56) to larger ones (ResNet-18, ResNet-34) and VGG-19, demonstrating its general applicability.

Key Results

  • Significant Compression: RSDTR achieved substantial parameter compression ratios (PCR) and FLOPS compression ratios (FCR) across various CNN architectures and datasets. For instance, on ResNet-20, it outperformed other pruning and low-rank methods in terms of PCR, FCR, and accuracy drop.
  • Competitive Accuracy: The compressed networks maintained classification accuracy comparable to their baseline counterparts. In many cases, RSDTR achieved higher accuracy than other low-rank methods for similar compression levels.
  • Reduced FLOPS Increase: Unlike some tensorized TR methods where FLOPS can increase, RSDTR demonstrated a reduction in FLOPS, making it more suitable for real-time applications.
  • Fast Compression Time: The compression process itself was found to be relatively fast, with times generally under 97 seconds on a high-end CPU, which is marginal compared to the fine-tuning process.

Limitations and Future Work

  • Exhaustive Search: While effective, the exhaustive search for optimal circular permutations and ranks can be computationally intensive, though the paper notes it's still faster than retraining.
  • Higher-Order CNNs: The authors suggest extending the method to compress higher-order CNNs where kernels are represented by higher-order tensors.
  • Hybrid Compression: Future work could explore combining RSDTR with other compression techniques like pruning to achieve even greater compression.
  • Tensor Network Structures: Investigating more advanced tensor network structures beyond TR decomposition for optimal kernel representation is also proposed.

Summary updated

BibTeX
@article{gabor2026reduced,
  author = {Mateusz Gabor and Rafal Zdunek},
  title = {{Reduced storage direct tensor ring decomposition for convolutional neural networks compression}},
  journal = {Neural Networks},
  year = {2026},
  doi = {10.1016/j.neunet.2025.107994},
}
Figure