Weight Sharing for On the Fly Model Scaling

This direction is split into other directions: dynamic model scaling. Its last version stays readable and citable here.

Version 1 of 197 papers113 conceptsfrozen 2026-10-04

Deep learning deployment usually faces a "Goldilocks" problem. A model trained for a high-end GPU is too heavy for a smartphone, while a model optimized for a smartphone lacks the accuracy required for server-side analysis. Traditionally, researchers solved this by training a suite of independent models—one for each hardware profile.

However, a growing body of research suggests that we don't need separate models. Instead, we can train a single elastic or slimmable neural network that can be sliced into different sizes on the fly. By sharing a single set of weights across multiple computational scales, these architectures allow for instant trade-offs between accuracy, latency, and power consumption.

The Intuition of Nested Weights

The core mechanism behind this research direction is weight sharing across nested sub-networks. Imagine a standard convolutional layer with 256 channels. In a slimmable network, as proposed by Jiahui Yu in Slimmable Neural Networks, we don't treat these channels as an unordered set. Instead, we treat them as a nested hierarchy.

If the device has full power, it uses all 256 channels. If the battery is low or the workload spikes, the system "slims" the model by only executing the first 64 or 128 channels. Because the smaller sub-networks are subsets of the larger ones, they share the same underlying parameters. This "once-for-all" approach, further explored in Slimmable Generative Adversarial Networks by Liang Hou, ensures that a single model file can serve a spectrum of hardware constraints without the memory overhead of storing multiple versions.

Solving the Feature Shift with Switchable Normalization

The primary challenge of sharing weights across different widths is that the internal statistics of the network change. Batch Normalization (BN) layers track the mean and variance of activations to keep training stable. However, the mean and variance of a layer with 64 active channels are fundamentally different from those of the same layer with 256 channels.

To solve this, Jiahui Yu introduced Switchable Batch Normalization (S-BN). While the convolutional weights are shared across all widths, the model maintains independent BN parameters for each predefined width. This allows the model to "re-calibrate" its normalization statistics depending on which sub-network is currently active. This simple architectural tweak is what allows a single set of weights to remain highly accurate across vastly different scales.

Optimization via Sandwich Training and Distillation

Training an elastic model is more complex than training a static one because the weights must satisfy multiple objectives simultaneously. If you only optimize for the largest width, the smaller sub-networks will perform poorly.

The research community has converged on two key training strategies to ensure consistency:

  1. Sandwich Training: In each training step, the model is updated using the largest width, the smallest width, and a few random intermediate widths. This ensures that the weights are robust across the entire range of possible configurations.
  2. In-place Distillation: The largest version of the model (the "teacher") generates soft targets for the smaller versions (the "students") during training. As shown in DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers, this allows the smaller sub-networks to inherit the rich representational knowledge of the full-scale model, often leading them to outperform models of the same size trained in isolation.

Beyond Channels: Elasticity in Depth and Attention

While early work focused on channel width, recent research has extended elasticity to every dimension of neural architectures.

  • Depth Elasticity: Models like Sorted LLaMA and the work of Benyou Wang on On Elastic Language Models allow Transformers to skip layers dynamically. By training intermediate layers to produce valid outputs, a single LLM can scale its depth to maintain stable latency during traffic spikes.
  • Attention and Resolution: Frameworks like Scala, introduced by Xi Chen in Slicing Vision Transformer for Flexible Inference, apply slicing to Vision Transformers, adjusting the number of attention heads and embedding dimensions.
  • Temporal Elasticity: In diffusion models, Shuai Yang demonstrated in Denoising Diffusion Step-aware Models that not every denoising step requires the full model. By using slimmable U-Nets, they can switch to smaller sub-networks for less critical timesteps, saving up to 76% of computation.

Why This Works: Regularization and Efficiency

The surprising finding across these papers is that slimmable training often acts as a form of regularization. Because the weights must be useful in multiple contexts (both in a small, constrained model and a large, expressive one), they are forced to learn more general and robust features.

This architectural flexibility is becoming essential for real-world systems. Whether it is preventing thermal throttling on mobile devices (Play It Cool), managing precise bitrates in video compression (Chenhao Zhang et al., Learned Rate Control), or enabling private "oblivious" inference on heterogeneous clients (Mohammad Samragh, On the Application of Binary Neural Networks in Oblivious Inference), elastic weight sharing provides a path toward AI that adapts to its environment rather than being limited by it.

Papers in this version

Showing 60 of 97 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_weight_sharing_for_on_the_fly_model_scal_0bebc3a6,
  title = {Weight Sharing for On the Fly Model Scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-sharing-for-on-the-fly-model-scaling/rdr_0bebc3a6b9ce4384920be8f7b04dbc21},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_weight_sharing_for_on_the_fly_model_scal_0bebc3a6_v1,
  title = {Weight Sharing for On the Fly Model Scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/weight-sharing-for-on-the-fly-model-scaling/rdr_0bebc3a6b9ce4384920be8f7b04dbc21/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}