Compact and Optimal Deep Learning with Recurrent Parameter Generators

Jiayun Wang, Yubei Chen, Stella X. Yu, Brian Cheung, Yann LeCun

Abstract

Deep learning has achieved tremendous success by training increasingly large models, which are then compressed for practical deployment. We propose a drastically different approach to compact and optimal deep learning: We decouple the Degrees of freedom (DoF) and the actual number of parameters of a model, optimize a small DoF with predefined random linear constraints for a large model of an arbitrary architecture, in one-stage end-to-end learning. Specifically, we create a recurrent parameter generator (RPG), which repeatedly fetches parameters from a ring and unpacks them onto a large model with random permutation and sign flipping to promote parameter decorrelation. We show that gradient descent can automatically find the best model under constraints with in fact faster convergence. Our extensive experimentation reveals a log-linear relationship between model DoF and accuracy. Our RPG demonstrates remarkable DoF reduction, and can be further pruned and quantized for additional run-time performance gain. For example, in terms of top-1 accuracy on ImageNet, RPG achieves 96% of ResNet18’s performance with only 18% DoF (the equivalent of one convolutional layer) and 52% of ResNet34’s performance with only 0.25% DoF! Our work shows significant potential of constrained neural optimization in compact and optimal deep learning.

Introduction: The Over-parameterization Dilemma

The current paradigm in deep learning follows a "bigger is better" philosophy, resulting in models with billions of parameters. While effective, these models are notoriously over-parameterized, containing far more independent variables than necessary for their target tasks. The standard industry workflow attempts to mitigate this by training a massive model and then applying post-hoc compression techniques like pruning or quantization.

This paper argues that this workflow is fundamentally backward. Instead of compressing a large model after the fact, can we optimize for compactness from the very beginning? The authors introduce Recurrent Parameter Generators (RPG) to decouple a model's architectural complexity (the number of layers and neurons) from its actual Degrees of Freedom (DoF)—the number of independent variables the optimizer actually tunes.

Methodology: Recurrent Parameter Generators (RPG)

The core innovation is the Recurrent Parameter Generator (RPG), which treats model weights as a constrained linear projection of a much smaller parameter set.

The Model Ring and Unpacking

Instead of assigning unique weights to every layer, the system maintains a single, small "ring" of free parameters ($W$). To populate the large model architecture, the RPG "fetches" parameters from this ring. To prevent layers from learning redundant features, the RPG applies random permutations and sign flipping during the unpacking process. This "destructive" weight sharing helps decorrelate the weights, allowing the model to retain high capacity even with minimal free parameters.

The RPG mechanism fetches parameters from a small Model Ring and unpacks them into a large Model Parameter space using random generators.

Linearly Constrained Optimization

Mathematically, the weights of the large model ($\hat{W}$) are defined as $\hat{W} = GW$, where $G$ is a fixed, random projection matrix. During training, the optimizer only updates the small set $W$. This approach allows for multi-scale sharing, where RPGs can be applied globally across an entire network or locally within specific blocks.

Key Insights and Theoretical Findings

Faster Convergence and Better Generalization

Contrary to intuition, optimizing in a smaller, constrained subspace often speeds up training. RPG models frequently reach peak accuracy faster than their unconstrained "vanilla" counterparts. Furthermore, because these models have fewer "knobs" to turn, they are less prone to memorizing training data (overfitting), leading to a smaller gap between training and validation accuracy and superior performance on out-of-distribution benchmarks like ObjectNet.

The Log-Linear Relationship

A significant discovery of this research is the clear log-linear (power-law) relationship between a model's DoF and its accuracy. This relationship provides a predictable framework for researchers to estimate performance trade-offs when shrinking a model before training begins.

Experimental Results

The RPG approach was validated across diverse tasks, including image classification (ImageNet, CIFAR), pose estimation (MPII), and multitask regression (S3DIS).

A comprehensive overview showing (c) the intuition of constrained optimization, (d) the log-linear DoF-accuracy relationship, and (e) RPG's superior performance compared to other compression methods.
  • ImageNet Efficiency: An RPG-based ResNet18 achieved 96% of the vanilla model's performance using only 18% of the DoF. More extremely, a ResNet34 reached 52% accuracy with just 0.25% of its original DoF.
  • Comparison to SOTA: RPG consistently outperformed established compression methods such as Knapsack, TAS, FPGM, and PruneNet at equivalent parameter counts.
  • Inference and Storage: RPGs are 15-25x faster during inference than Deep Equilibrium Models (DEQ) because they avoid iterative solvers. For a standard ResNet18, the RPG approach reduced the model file size by 79% with only a 2% drop in accuracy.

Conclusion

Recurrent Parameter Generators represent a shift toward constrained neural optimization. By decoupling architectural depth from degrees of freedom, RPGs enable the creation of models that are both compact and optimal. Future work may explore non-linear parameter generators and hardware-specific kernels to further capitalize on the efficiency of the "unpacking" process.

Summary updated

Abstract

Deep learning has achieved tremendous success by training increasingly large models, which are then compressed for practical deployment. We propose a drastically different approach to compact and optimal deep learning: We decouple the Degrees of freedom (DoF) and the actual number of parameters of a model, optimize a small DoF with predefined random linear constraints for a large model of an arbitrary architecture, in one-stage end-to-end learning.Specifically, we create a recurrent parameter generator (RPG), which repeatedly fetches parameters from a ring and unpacks them onto a large model with random permutation and sign flipping to promote parameter decorrelation. We show that gradient descent can automatically find the best model under constraints with in fact faster convergence.Our extensive experimentation reveals a log-linear relationship between model DoF and accuracy. Our RPG demonstrates remarkable DoF reduction, and can be further pruned and quantized for additional run-time performance gain. For example, in terms of top-1 accuracy on ImageNet, RPG achieves 96% of ResNet18’s performance with only 18% DoF (the equivalent of one convolutional layer) and 52% of ResNet34’s performance with only 0.25% DoF! Our work shows significant potential of constrained neural opti-mization in compact and optimal deep learning.
BibTeX
@inproceedings{wang2023compact,
  author = {Jiayun Wang and Yubei Chen and Stella X. Yu and Brian Cheung and Yann LeCun},
  title = {{Compact and Optimal Deep Learning with Recurrent Parameter Generators}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=ycgoSpE1lQ},
}
Figure