Nested Subnetworks for Dynamic Hardware Scaling

95 concepts77 papers1 proposal

Deploying deep learning models to edge devices—such as smartphones, IoT sensors, and embedded cameras—presents a fundamental dilemma. Hardware resources fluctuate constantly due to battery levels, thermal throttling, and competing background applications.

The traditional solution to this problem is to train a suite of discrete models spanning different sizes (e.g., MobileNet for low-power states and ResNet-101 for high-power states). However, this approach introduces a massive storage overhead and severe latency spikes when swapping models in and out of memory. As noted by Mi Zhang 0002 et al. in NestDNN, structuring neural networks as nested subsets eliminates this overhead entirely, allowing for instantaneous performance scaling without reloading model files into memory.

The core idea behind this research direction is to embed multiple valid sub-networks of varying capacities inside a single "supernet" using a single set of shared weights. Like a Russian Matryoshka doll, a smaller model is simply a strict subset of a larger one.

The Intuition of Weight-Shared Slicing

To understand why this works, consider a standard convolutional layer. The layer transforms an input tensor with $C_{in}$ channels into an output tensor with $C_{out}$ channels using a weight matrix. In a slimmable architecture, we can execute only a fraction of these channels on the fly.

As demonstrated by Fei Yang et al. in Slimmable Compressive Autoencoders, executing a subset of active channels within a single weight matrix allows a model to operate at multiple computational complexities. If the hardware is constrained, the system slices the weight tensor, utilizing only the first $k$ channels ($k < C_{out}$). The early channels are forced to capture the most critical, coarse features of the data, while the outer channels encode residual, high-fidelity details.

Master Weight Matrix (C_out x C_in)
+-----------------------+--------+
|                       |        |
|  Sub-network Weights  |        |  <- Active at 50% Width
|     (First 50%)       |        |
|                       |        |
+-----------------------+--------+
|                                |
|      Remaining Channels        |  <- Activated only at 100% Width
|                                |
+--------------------------------+

However, you cannot simply take a standard pre-trained network and drop half its channels; doing so destroys its predictive power. As Jiahui Yu et al. observe in Universally Slimmable Networks and Improved Training Techniques, slimmability must be explicitly optimized during training because feature distributions shift drastically across different widths.

The primary culprit behind this failure is Batch Normalization (BN). A single set of running mean and variance statistics cannot accurately normalize activations across multiple widths simultaneously. To resolve this optimization conflict, Shengsen Wu et al. in Switchable Representation Learning Framework with Self-Compatibility utilize independent batch normalization layers for each sub-model configuration. By maintaining distinct BN statistics while sharing the underlying convolutional weights, the network successfully preserves functional feature representations across all computational scales.

Co-optimizing the Continuum: The Sandwich Rule

Training a single set of weights to satisfy multiple capacities simultaneously sounds like an under-constrained problem. If we optimize for the smallest sub-network, we might degrade the performance of the largest. If we optimize only for the largest, the smaller subsets become unusable.

To solve this, the field relies on a training paradigm known as the sandwich rule, formalized by Jiahui Yu et al.. During each training step, gradients are computed and accumulated for:

  1. The largest possible sub-network (the upper bound).
  2. The smallest possible sub-network (the lower bound).
  3. A few randomly sampled intermediate sub-networks.

By bounding the optimization space from both ends, the entire continuum of intermediate weights is regularized.

Furthermore, to ensure that the smaller sub-networks behave compatibly with the larger ones, the authors introduce inplace distillation. During training, the largest sub-network generates soft targets (predictions) for a given batch. The smaller sub-networks are then trained to predict these soft targets rather than relying solely on the hard ground-truth labels. This acts as an on-the-fly knowledge transfer mechanism, ensuring that feature representations remain mutually compatible across all scales.

Expanding Dimensions: Depth, Attention, and State Spaces

While early iterations of this concept focused primarily on channel width in convolutional networks, the underlying principle of nested weight sharing has expanded across multiple architectural dimensions and model families.

Depth Scaling

Instead of just dropping channels, architectures can dynamically scale their depth by skipping layers. Lu Hou et al. demonstrate this multi-dimensional flexibility in DynaBERT: Dynamic BERT with Adaptive Width and Depth. By training a single model to support multiple width ratios and layer depths during distillation, they provide sub-networks that can instantly adapt their computational footprint to meet varying latency constraints.

Attention Heads

In Vision Transformers (ViTs), the nesting strategy can be applied directly to the multi-head attention mechanism. In ThinkingViT: Matryoshka Thinking Vision Transformer for Elastic Inference, Ali Hojjat et al. order attention heads according to importance. This allows a single backbone to activate only a small subset of heads for simple inputs and expand to the full set for complex inputs, achieving massive compute savings on average.

State Space Models

The Matryoshka paradigm has also proven effective in modern non-transformer architectures. Abhinav Shukla et al. introduce this concept to State Space Models (SSMs) in MatMamba: A Matryoshka State Space Model. By independently selecting different nested dimensions for each layer, a single trained model can be deployed across hundreds of distinct, high-performing submodel sizes without losing embedding compatibility.

By decoupling training complexity from inference deployment constraints, slimmable and nested architectures transform neural networks from rigid, static structures into fluid, resource-aware computations.

Key Foundational Work

NestedNet: Learning Nested Sparse Structures in Deep Neural Networks (2018) — contributed 2 concept elements to this direction.
Slimmable Neural Networks (2018) — contributed 1 concept elements to this direction.
NestDNN: Resource-Aware Multi-Tenant On-Device Deep Learning for Continuous Mobile Vision (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_model_scaling_72128dc7,
  title = {dynamic model scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-model-scaling/rdr_72128dc7265f446fa36557bad860df0d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_model_scaling_72128dc7_v1,
  title = {dynamic model scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-model-scaling/rdr_72128dc7265f446fa36557bad860df0d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake