Compact and Optimal Deep Learning with Recurrent Parameter Generators

Jiayun Wang, Yubei Chen, Stella X. Yu, Brian Cheung, Yann LeCun

TL;DR

Instead of the traditional "train big, then compress" pipeline, this paper proposes directly optimizing a small set of free parameters that are "unpacked" into a large model architecture using a Recurrent Parameter Generator (RPG). This approach achieves state-of-the-art compression ratios while actually speeding up training and improving model generalization.

What Problem Does This Paper Address?

Modern deep learning is defined by a "bigger is better" mantra. Models like GPT-3 boast hundreds of billions of parameters, far exceeding the amount of training data available. This massive over-parameterization is great for performance but terrible for deployment on edge devices with limited memory and power.

The standard solution is a two-stage process: train a giant model, then use techniques like pruning or quantization to squeeze it down. The authors of this paper argue that this is fundamentally backwards. They ask: Can we decouple a model's architecture from its actual Degrees of Freedom (DoF)? In other words, can we have the representational power of a deep, complex network while only needing to learn and store a tiny fraction of its parameters?

Methods

The core innovation is the Recurrent Parameter Generator (RPG). Here’s how it works:

  1. The Model Ring: Instead of independent weights for every layer, the model maintains a small, fixed set of free parameters $W$ arranged in a "ring."
  2. Unpacking with Constraints: To populate the weights of a large architecture (like a ResNet), the RPG repeatedly fetches parameters from this ring.
  3. Destructive Weight Sharing: If you just copied the same weights to every layer, the model wouldn't learn much. To prevent this, the RPG applies random permutations and sign flips (fixed, predefined linear constraints) as it "unpacks" the parameters. This ensures that even though weights are shared, they are decorrelated across different layers.
  4. One-Stage Optimization: Unlike pruning, there is no "pre-training." Gradient descent directly optimizes the small set of free parameters in the ring. The large model is essentially a "constrained" version of the architecture.
Figure 1: We propose a novel approach to compact and optimal deep learning by decoupling model DoF and model parameters. a) Existing methods first finds the optimal in a large model space and then compress it for practical deployment. b) We propose to start with a small (DoF) model of free parameters, use recurrent parameter generator (RPG) to unpack them onto a large model with predefined random linear projections. c) Gradient descent finds the optimal model of a small DoF under these linear constraints with faster converge than training the large unpacked model itself (Fig.5b). If the DoF is too small, the optimal large model may fall out of the constrained subpsace. However, at a sufficiently large DoF, RPG gets rid of redundancy and often finds a model with little loss in accuracy. d) RPG reveals a log-linear relationship between model DoF and accuracy. e) RPG achieves the same ImageNet accuracy with half of the ResNet-vanilla DoF. RPG also outperforms other state-of-the-art compression approaches.

Datasets

The authors validated RPG across a diverse set of tasks:

  • Image Classification: ImageNet, CIFAR-10, and CIFAR-100.
  • Human Pose Estimation: MPII Human Pose dataset.
  • Multi-task Regression: S3DIS (predicting depth and surface normals).
  • Out-of-Distribution (OOD) Evaluation: ObjectNet, to test how well the compressed models generalize to new data.

Key Insights and Ideas

  • The Log-Linear Law: The researchers discovered a predictable, log-linear relationship between a model's DoF and its accuracy. This "scaling law" allows researchers to estimate how much they can compress a model before performance drops off a cliff.
  • Faster Convergence: Counter-intuitively, optimizing a small DoF under linear constraints is faster than training the full, unconstrained model. The constraints seem to guide gradient descent through a more efficient search space.
  • Generalization over Overfitting: Because RPG models have fewer "free" parameters to wiggle, they are much less likely to overfit the training data. This results in a smaller gap between training and validation performance and significantly better results on out-of-distribution datasets like ObjectNet.
  • Architecture vs. Parameters: RPG proves that the structure of a network (its depth and connectivity) is often more important than having unique values for every single weight.
RPG demonstrates a log-linear relationship between DoF and accuracy and converges faster than vanilla models.

Key Results

  • Extreme Compression: On ImageNet, an RPG-based ResNet18 achieves 96% of the vanilla model's accuracy using only 18% of the DoF (roughly the equivalent of a single convolutional layer).
  • Efficiency: A ResNet34-RPG matched the accuracy of a vanilla ResNet34 while using 50% fewer backbone parameters.
  • Outperforming SOTA: RPG consistently beat other state-of-the-art compression methods like Knapsack pruning, Deep Compression, and Lego filters when compared at the same DoF.
  • Quantization Ready: RPG models are highly robust; 8-bit quantization resulted in a negligible accuracy drop of only 0.1%.

Limitations and Future Work

While RPG is highly effective at reducing the number of parameters (storage), it doesn't inherently reduce the number of computations (FLOPs) or inference time, as the full large architecture still needs to be executed. The authors note that RPG can be combined with coarse-grained pruning to address this.

Future work could explore non-linear parameter generators. This paper stuck to random linear projections for simplicity and efficiency, but more complex, learnable unpacking functions might unlock even higher levels of "optimal" deep learning.

Summary updated

Abstract

Deep learning has achieved tremendous success by training increasingly large models, which are then compressed for practical deployment. We propose a drastically different approach to compact and optimal deep learning: We decouple the Degrees of freedom (DoF) and the actual number of parameters of a model, optimize a small DoF with predefined random linear constraints for a large model of arbitrary architecture, in one-stage end-to-end learning. Specifically, we create a recurrent parameter generator (RPG), which repeatedly fetches parameters from a ring and unpacks them onto a large model with random permutation and sign flipping to promote parameter decorrelation. We show that gradient descent can automatically find the best model under constraints with faster convergence. Our extensive experimentation reveals a log-linear relationship between model DoF and accuracy. Our RPG demonstrates remarkable DoF reduction and can be further pruned and quantized for additional run-time performance gain. For example, in terms of top-1 accuracy on ImageNet, RPG achieves $96\%$ of ResNet18's performance with only $18\%$ DoF (the equivalent of one convolutional layer) and $52\%$ of ResNet34's performance with only $0.25\%$ DoF! Our work shows a significant potential of constrained neural optimization in compact and optimal deep learning.
BibTeX
@inproceedings{wang2023compact,
  author = {Jiayun Wang and Yubei Chen and Stella X. Yu and Brian Cheung and Yann LeCun},
  title = {{Compact and Optimal Deep Learning with Recurrent Parameter Generators}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=ycgoSpE1lQ},
}
Figure