Cross Scale Parameter Sharing in Hierarchical Architectures

This direction is split into other directions: cross-scale weight sharing. Its last version stays readable and citable here.

Version 1 of 1197 papers265 conceptsfrozen 2026-10-04

In traditional deep learning, we often treat different scales of data as distinct problems. If we want to detect a large car and a small car, we typically pass the image through a series of layers where each layer is responsible for a specific level of detail. However, this approach leads to a massive explosion in parameters.

The core insight of this research direction is that the logic of the world is often scale-invariant. An edge is an edge, and a motion pattern is a motion pattern, whether it spans five pixels or fifty. By sharing a single set of weights across different resolutions or hierarchical levels, researchers are building models that are significantly more compact, faster to train, and more robust to changes in scale.

The Pyramid Recurrent Mechanism

The most direct way to implement this idea is through a "pyramid" structure where the same neural module is applied iteratively to different resolutions of the same input. Instead of having ten different layers for ten different scales, you have one "brain" that looks at the image at multiple zoom levels.

In PRAFlow_RVC, Zhexiong Wan and colleagues demonstrate this by using a single recurrent unit with shared weights across a resolution pyramid for optical flow. Similarly, Shu Kong and Charless Fowlkes developed mgPFF (Multigrid Predictive Filter Flow), which uses shared weights across resolutions to track motion.

The intuition here is simple: if a model learns how to track a pixel's movement at a coarse resolution, that same logic should apply when refining the movement at a fine resolution. This "coarse-to-fine" strategy, powered by weight sharing, allows Xin Jin 0023 to achieve state-of-the-art video frame interpolation in UPR-Net with only 1.7 million parameters—a fraction of the size of competing models.

Weight Inheritance and Dilation Locking

Beyond simply resizing the input, researchers have found ways to share parameters within the convolutional filters themselves.

In Learning Versatile Filters for Efficient Convolutional Neural Networks, Yunhe Wang introduces "weight inheritance." Instead of training separate $3\times3$ and $5\times5$ filters, the model treats the $3\times3$ filter as a subset of the $5\times5$ one. This forces the model to capture multi-scale features at zero additional storage cost.

A similar concept appears in DetectoRS, where Siyuan Qiao uses "Switchable Atrous Convolution." Here, convolutions with different dilation rates (which change the receptive field size) share the same base weights. By "locking" these weights together, the model learns a unified representation that is effective across multiple spatial resolutions, making it much easier to fine-tune pre-trained models for complex tasks like object detection.

Scale Invariance in Transformers and NeRFs

The move toward Transformers hasn't changed this fundamental goal; it has only changed the mechanism. In A free lunch from ViT, Yuan Zhang shows that a single shared encoder can process both global images and local crops. This forces the Transformer to learn features that are useful at both holistic and granular levels, which is essential for fine-grained recognition (like distinguishing between two very similar species of birds).

This concept even extends to 3D scene representation. Ben Mildenhall and the team behind Mip-NeRF solved the problem of aliasing by encoding scale directly into the input features. This allows a single neural network to represent a scene at any scale. Because the same weights generate both the zoomed-out and zoomed-in views, the representations are perfectly consistent, leading to sharper renders and a 50% reduction in model size.

Beyond Pixels: Graphs and Sequences

The principle of cross-scale sharing is not limited to computer vision. It is a general strategy for any data with hierarchical structure:

  • Graphs: Wenhao Zhu shares weights across transformer layers operating at different levels of a graph hierarchy in Hierarchical Transformer for Scalable Graph Learning, exploiting the self-similarity of structural patterns.
  • Algorithms: Alex Nowak uses identical neural modules at every level of a recursive "divide and conquer" hierarchy in DiCoNet, ensuring that logic learned for small problems generalizes to massive ones.
  • Medical Imaging: In Meta Segmentation Network, Tong Wu uses weight sharing across resolution branches to segment massive whole-slide images, reducing parameters by 80% while maintaining local detail.

Why It Works: The Self-Similarity Inductive Bias

Why does sharing weights across scales work so well? It acts as a powerful inductive bias.

By forcing a model to use the same parameters for a small version of an object and a large version, we are teaching it that "object-ness" is independent of size. This serves as a form of regularization, preventing the model from overfitting to the specific noise present at a single resolution. As Tong Wu notes, this is particularly vital in fields like medical imaging where data is scarce.

Ultimately, cross-scale parameter sharing moves us away from "brute force" architectures that need a new layer for every new detail, toward more elegant, "fractal" models that understand the underlying self-similarity of the world.

Papers in this version

Showing 60 of 197 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_cross_scale_parameter_sharing_in_hierarc_318d5c36,
  title = {Cross Scale Parameter Sharing in Hierarchical Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-scale-parameter-sharing-in-hierarchical-architectures/rdr_318d5c36c95147829b9d047ec5f966f9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_cross_scale_parameter_sharing_in_hierarc_318d5c36_v1,
  title = {Cross Scale Parameter Sharing in Hierarchical Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/cross-scale-parameter-sharing-in-hierarchical-architectures/rdr_318d5c36c95147829b9d047ec5f966f9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}