How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?

Jetze Schuurmans, kim batselier, Julian Kooij

Compressing large neural networks using Tensor Decompositions (TD)—such as CP, Tucker, or Tensor Train—is a powerful technique to reduce parameter counts and computational overhead. However, the process introduces a "hyperparameter explosion." For every layer in a network, a researcher must decide which layer to compress, which decomposition method to use, and what compression rank (level) to target.

Evaluating the full model's accuracy for every possible combination is computationally ruinous. To bypass this, researchers often rely on a proxy metric: they assume that if the compressed weights are mathematically close to the original weights (i.e., a low approximation error), the model's performance will remain high.

Surprisingly, prior to this research, the validity of this assumption had not been systematically verified across different layers, decomposition methods, and fine-tuning stages. This paper investigates whether the approximation error actually predicts how much a neural network's accuracy will drop, revealing that while certain metrics are reliable, their efficacy depends heavily on the compression level and model architecture.

Experimental Design

The authors conducted an extensive empirical study to measure the correlation (using Kendall’s $\tau$ rank correlation) between various error metrics and the actual performance drop of the model.

The Variables Tested: * Decomposition Methods: CP, Tucker, and Tensor Train (TT). * Error Metrics: Six variations were tested by crossing two targets (Weights vs. Features) with three normalization types (Absolute, Relative, and Scaled). * Fine-tuning: Performance was measured both immediately after compression and after a "recovery" fine-tuning phase. * Models & Datasets: ResNet-18 and GaripovNet (a 7-layer CNN) evaluated on CIFAR-10 and Fashion-MNIST. A dedicated validation set was used for monitoring early stopping and hyperparameter selection to prevent overfitting to the test set.

The core methodology involved calculating how often a "lower error" actually leads to "better performance" across hundreds of experimental runs.

Key Insights: Choosing the Right Metric

One of the most critical findings of the study is that not all approximation errors are created equal.

Correlation of different approximation error metrics with performance drop, before and after fine-tuning.
  • Relative is King: The Relative Weight Error (the Frobenius norm of the error divided by the norm of the original weights) was the most consistent predictor of performance. As seen in the data, Absolute Error was a terrible proxy, particularly for ResNet-18, because it fails to account for the varying scales and importance of different layers.
  • Features Aren't Worth the Fuss: Previous literature suggested that measuring error on "features" (the actual output of a layer given some data) would be a better proxy than just looking at weights. However, this paper finds that feature-based metrics do not significantly improve correlation. This is good news for practitioners: you can likely stick to weight-based metrics and save the extra computation of a forward pass.
  • Comparing Methods vs. Layers: The approximation error is much more useful for choosing which method to use for a specific layer (e.g., "Should I use CP or Tucker here?") than it is for deciding which layer to compress first.

The Impact of Compression Levels and Fine-Tuning

The reliability of the approximation error proxy is not static; it shifts dramatically depending on how aggressively the model is being compressed.

Correlation between approximation error and performance across different compression levels.
  • Compression Level Matters: At low compression rates (reducing parameters by only 10-25%), the approximation error is a noisy, unreliable signal. The proxy only becomes truly informative at higher compression levels (50% and above), where the structural impact on the model is significant enough to rise above standard training noise.
  • Positive Correlation: In almost all cases (except Absolute Weight error on ResNet), there was a significant positive correlation between approximation error and performance error.
  • Fine-tuning Impact: While fine-tuning generally improves overall model performance, it can sometimes weaken the correlation for certain architectures. For example, in GaripovNet, CP-decomposed layers were harder to optimize, causing the correlation to drop after fine-tuning compared to Tucker or TT.

Limitations and Future Work

The authors acknowledge a few boundaries to their study. First, the research focused exclusively on convolutional layers. While the logic should theoretically hold for fully connected layers (by reshaping them into tensors), this was not explicitly tested. Second, the experiments evaluated the error after a single compression step. In practice, many pipelines use iterative compression (compress, fine-tune, repeat), and it remains unclear if the proxy remains as informative over many cycles. Finally, while the study covered the "Big Three" decompositions (CP, Tucker, TT), emerging hybrid approaches remain unexplored.

Conclusion: If you are building an automated tool to compress neural networks, use Relative Weight Error to select your decomposition method. It is fast, does not require passing data through the network, and provides a genuinely useful signal for maintaining model accuracy.

Summary updated

Abstract

Tensor decompositions have been successfully applied to compress neural networks. The compression algorithms using tensor decompositions commonly minimize the approximation error on the weights. Recent work assumes the approximation error on the weights is a proxy for the performance of the model to compress multiple layers and fine-tune the compressed model. Surprisingly, little research has systematically evaluated which approximation errors can be used to make choices regarding the layer, tensor decomposition method, and level of compression. To close this gap, we perform an experimental study to test if this assumption holds across different layers and types of decompositions, and what the effect of fine-tuning is. We include the approximation error on the features resulting from a compressed layer in our analysis to test if this provides a better proxy, as it explicitly takes the data into account. We find the approximation error on the weights has a positive correlation with the performance error, before as well as after fine-tuning. Basing the approximation error on the features does not improve the correlation significantly. While scaling the approximation error commonly is used to account for the different sizes of layers, the average correlation across layers is smaller than across all choices (i.e. layers, decompositions, and level of compression) before fine-tuning. When calculating the correlation across the different decompositions, the average rank correlation is larger than across all choices. This means multiple decompositions can be considered for compression and the approximation error can be used to choose between them.
BibTeX
@inproceedings{schuurmans2023how,
  author = {Jetze Schuurmans and kim batselier and Julian Kooij},
  title = {{How Informative is the Approximation Error from Tensor Decomposition for Neural Network Compression?}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=sKHqgFOaFXI},
}
Figure