Scalable Weight Reparametrization for Efficient Transfer Learning

Byeonggeun Kim, Jun-Tae Lee, Seunghan Yang, Simyung Chang

Abstract

Transfer learning is essential for deploying deep learning models on edge devices, yet existing methods often force a trade-off between accuracy, storage (parameter efficiency), and latency (computational efficiency). This paper introduces Scalable Weight Reparametrization (SWR), a novel transfer learning framework that utilizes a policy network to dynamically select layer-wise updates based on a user-defined parameter budget. By reparametrizing weights such that task-specific deltas can be folded into the base model, SWR achieves state-of-the-art accuracy with zero additional inference overhead. We demonstrate SWR's effectiveness across audio and vision tasks, showing it often outperforms full fine-tuning while using a fraction of the parameters.

Introduction

Modern AI relies heavily on transfer learning, where a large-scale pre-trained model is adapted to a specific downstream task. However, deploying these models on resource-constrained edge devices—such as smartwatches or IoT sensors—presents a significant challenge. Practitioners typically face three suboptimal choices:

  1. Full Fine-tuning: High accuracy, but requires storing a complete set of weights for every task, leading to massive storage requirements.
  2. Adapter Modules: High parameter efficiency, but introduces new layers that increase latency during inference.
  3. Selective Tuning: Updating only a subset of layers, which often requires manual, heuristic-based selection to meet specific hardware constraints.

Qualcomm AI Research's Scalable Weight Reparametrization (SWR) addresses these issues by automating the selection of layers to update, allowing for a controllable trade-off between accuracy and parameter count without any computational penalty during inference.

Methodology

Weight Reparametrization and Folding

The core innovation of SWR is its approach to weight updates. Instead of adding new architectural components, SWR expresses the weight of any given layer $l$ as: $$w_l = w_l^0 + b_l w'_l$$ Where:

  • $w_l^0$ represents the frozen, pre-trained weights.
  • $w'_l$ is a learnable, task-specific "delta."
  • $b_l \in \{0, 1\}$ is a binary decision from a policy network.

Because $w'_l$ has the same dimensionality as the original weights, the final learned parameters can be mathematically "folded" back into the base weights ($w_{final} = w_l^0 + w'_l$) after training. This ensures the model architecture remains unchanged, resulting in zero inference overhead.

The Policy Network

To determine which layers should receive updates ($b_l$), SWR employs a small Policy Network ($h$). This network takes a target cost $c$ (e.g., 0.1 for a 10% parameter budget) as an input. To make the discrete binary selection process differentiable for backpropagation, the researchers utilized the Gumbel-Softmax trick. This allows the system to learn an optimal selection policy that prioritizes layers providing the highest accuracy gains for the lowest "cost."

Two-Stage Training Process

To maintain stability, SWR uses a two-stage training regime:

  1. Policy Optimization: The policy network is trained to identify the most informative layers across various cost constraints by evaluating them against a "super-net" where all layers are potentially active.
  2. Weight Fine-tuning: Once the policy is established for a specific budget $c$, the policy is frozen, and only the selected task-specific deltas ($w'_l$) are trained.

Experimental Evaluation

Datasets and Benchmarks

SWR was evaluated on two distinct domains:

  • Multi-lingual Keyword Spotting (KWS): A benchmark combining the English Google Speech Commands (GSC2) with the Mandarin Chinese Scripted Speech Corpus (MC-KWS), using the BCResNet-8 backbone.
  • ImageNet-to-Sketch: A vision benchmark covering five diverse tasks: Flowers, WikiArt, Sketch, Cars, and CUBS.

Key Results

  • Superior Efficiency: On the BCResNet-8 backbone, SWR with a 10% parameter budget ($c=0.1$) outperformed established methods like SpotTune and Residual Adapters, despite those methods using 1.4x to 4x more parameters.
  • Regularization Effect: In several experiments, SWR achieved higher accuracy than full fine-tuning. This suggests that freezing the majority of pre-trained weights acts as a regularizer, preventing the model from overfitting on smaller downstream datasets.
  • Cross-Architecture Scalability: While optimized for "tiny" models, SWR scaled effectively to larger architectures like ResNet-50, achieving state-of-the-art results on the ImageNet-to-Sketch benchmark while maintaining a significantly smaller parameter footprint than competitors.

Discussion and Limitations

SWR's primary advantage is its ability to "dial in" efficiency. The policy network learns that updating "cheap" layers (like depth-wise convolutions) is often more efficient than updating "expensive" 1x1 convolutions.

However, the method does introduce additional training complexity due to the two-stage requirement. Furthermore, the current implementation operates at a layer-wise granularity. Future iterations could explore channel-level or weight-level policies to further refine the accuracy-to-parameter ratio, though this would increase the complexity of the policy network itself.

Conclusion

Scalable Weight Reparametrization offers a robust solution for deploying multi-task AI on the edge. By combining a trainable policy network with weight folding, SWR provides a flexible, automated way to meet strict parameter budgets without sacrificing inference speed or accuracy. This makes it a highly practical tool for the next generation of efficient, on-device machine learning.

Summary updated

Abstract

This paper proposes a novel, efficient transfer learning method, called Scalable Weight Reparametrization (SWR) that is efficient and effective for multiple downstream tasks. Efficient transfer learning involves utilizing a pre-trained model trained on a larger dataset and repurposing it for downstream tasks with the aim of maximizing the reuse of the pre-trained model. However, previous works have led to an increase in updated parameters and task-specific modules, resulting in more computations, especially for tiny models. Additionally, there has been no practical consideration for controlling the number of updated parameters. To address these issues, we suggest learning a policy network that can decide where to reparametrize the pre-trained model, while adhering to a given constraint for the number of updated parameters. The policy network is only used during the transfer learning process and not afterward. As a result, our approach attains state-of-the-art performance in a proposed multi-lingual keyword spotting and a standard benchmark, ImageNet-to-Sketch, while requiring zero additional computations and significantly fewer additional parameters.
BibTeX
@inproceedings{kim2023scalable,
  author = {Byeonggeun Kim and Jun-Tae Lee and Seunghan Yang and Simyung Chang},
  title = {{Scalable Weight Reparametrization for Efficient Transfer Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=_fSUUhS6H6t},
}
Figure