Scale Dependent Compressibility in Over Parameterized Models

115 concepts94 papers4 proposals

When we scale a neural network from one billion to one hundred billion parameters, it is tempting to assume that the larger model is simply storing one hundred times more unique, distinct concepts. However, empirical evidence reveals a more fascinating reality: as neural networks grow, their internal representation becomes disproportionately redundant.

This phenomenon—where parameter and activation redundancy scales non-linearly with model size—fundamentally alters how networks tolerate compression. Rather than making a model fragile, extreme over-parameterization creates a structural buffer that makes massive architectures uniquely compressible, allowing them to achieve high sparsity with virtually zero performance loss.

The Scaling Law of Redundancy

Why does a massive model tolerate losing half its weights better than a small model? The answer lies in how information is distributed across scales. In smaller networks, parameters are highly compact and tightly coupled; removing even a small fraction disrupts the fragile coordination required to propagate signals cleanly. As networks scale up, they develop an inherent cushion of low-utility and overlapping representations.

This was quantified by Filip Szatkowski et al. in Universal Properties of Activation Sparsity in Modern Large Language Models, who discovered that as language models grow in parameter count, their activation redundancy increases, and the effective rank of their activations decreases. In other words, larger models naturally rely on a smaller relative fraction of their capacity for any given computation, leaving a massive sea of redundant activations that can be safely zeroed out.

As a result, aggressive compression techniques that completely break smaller architectures work seamlessly on massive ones. For instance, Elias Frantar et al. demonstrated in SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot that 50-60% of weights can be stripped from a 175-billion-parameter model in a single pass without any retraining. Conversely, as Guanchen Li et al. noted in Enhancing One-Shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism, these exact same one-shot pruning heuristics cause catastrophic accuracy drops when applied to smaller, more compact models, which lack the internal redundancy required to absorb the shock of sudden weight removal.

The Optimization Scaffold: Large-Sparse vs. Small-Dense

A natural question arises: if a 14-billion-parameter model can be pruned by 50% to run at the size of a 7-billion-parameter model, why not simply train a 7-billion-parameter dense model from scratch?

The reason comes down to the geometry of the optimization landscape. Training a small, dense model forces the optimization algorithm to navigate a highly constrained, low-dimensional landscape riddled with poor local minima. In contrast, starting with an oversized, over-parameterized model provides a high-dimensional space with far more degrees of freedom. This extra capacity acts as an optimization scaffold, allowing gradient descent to easily discover robust feature representations and highly efficient sub-networks that are mathematically unreachable from a narrow starting point. Once the optimal pathways are established, the scaffolding can be carved away.

This core intuition is supported across multiple architectural domains:

Structural Resilience Across Components

This scale-dependent resilience is not limited to unstructured weight pruning; it manifests across entire structural components of modern architectures, including layers, modules, and attention mechanisms.

Depth and Layer Redundancy

As models grow deeper, individual layers begin to perform highly similar transformations, making them prime targets for structural removal. Dmitriy Shopkhoev et al. observed in ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization that larger language models exhibit a profound resilience to depth pruning, allowing entire blocks to be excised with minimal impact.

This layer-wise redundancy can even be exploited dynamically during inference. For example, Heming Xia et al. developed SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration, which accelerates inference by adaptively skipping intermediate layers on-the-fly to create a temporary draft model—a strategy that becomes significantly more effective as the base model's parameter count scales up.

Mixture-of-Experts and Modular Redundancy

In sparse conditional routing architectures like Mixture-of-Experts (MoE), scaling the number of experts inherently introduces low-utility components. Geng Zhang et al. demonstrated in MoNE: Replacing Redundant Experts with Lightweight Novices for Structured Pruning of MoE that larger MoE models possess a higher proportion of redundant experts. These can be safely replaced by lightweight constant vectors ("novices") without degrading capability.

Similarly, when compressing functional modules via matrix decomposition, Chi-Heng Lin et al. found in MoDeGPT: Modular Decomposition for Large Language Model Compression that larger architectures exhibit far less performance degradation than smaller ones at identical compression ratios.

Ultimately, this body of research shifts our understanding of model capacity. Over-parameterization is not a wasteful byproduct of scaling; it is a fundamental mechanism that reshapes the optimization landscape, providing the necessary redundancy to discover, isolate, and extract highly efficient, high-performing sparse structures.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_redundancy_scaling_50e325b3,
  title = {parameter redundancy scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-redundancy-scaling/rdr_50e325b33e024c3c8eb2ac5a52acd44b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameter_redundancy_scaling_50e325b3_v1,
  title = {parameter redundancy scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-redundancy-scaling/rdr_50e325b33e024c3c8eb2ac5a52acd44b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake