Global Optimization of Compressed Neural Architectures

This direction is split into other directions: greedy layer-wise optimization. Its last version stays readable and citable here.

Version 1 of 189 papers93 conceptsfrozen 2026-10-04

When we try to make a deep neural network smaller, the most intuitive approach is to look at it layer by layer. We might prune the least important filters in Layer 1, then move to Layer 2, and so on. This "greedy" strategy is computationally efficient, but it rests on a flawed assumption: that the layers of a neural network are independent components.

In reality, neural networks are highly integrated systems. A decision made in an early layer ripples through the entire architecture, changing the feature distributions and redundancy levels of every subsequent layer. A growing body of research suggests that to achieve true efficiency, we must move away from local, sequential tuning and toward global coordination.

The Failure of Greedy Optimization

The primary critique of traditional compression—as noted by researchers like Sukrit Shankar in Refining Architectures of Deep Convolutional Neural Networks—is that optimizing parameters on a layer-by-layer basis fails to find a global optimum. When we prune a single layer in isolation, we ignore the "interdependencies" between layers.

This problem is particularly acute in modern architectures like Mixture-of-Experts (MoE). Cheng Yang demonstrates in MoE-I² that pruning decisions for one expert layer can negatively impact the routing logic of the next. Because the data flow is coupled across the network, local optimization often leads to a "performance cliff" where the model's accuracy drops sharply because the cumulative error of these independent decisions becomes unmanageable.

Modeling the Ripple Effect

To solve this, researchers have begun modeling the network as a unified sequence rather than a stack of independent blocks. Ya Zhang introduced Recursive Bayesian Pruning, which treats the "ripple effect" of dropout noise across layers as a Markov chain. By accounting for how noise propagates, the method can prune channels more aggressively while maintaining the network's overall stability.

Similarly, Taiji Suzuki argues in Spectral-Pruning that a network's complexity is defined by its "statistical degrees of freedom" across its entire depth. If we only look at one layer, we might see high variance and assume it is important; however, a global spectral analysis might reveal that this variance is redundant given the information already captured by previous layers.

Global Rank and Sparsity Allocation

If layers are interdependent, then the "budget" for parameters or rank should be allocated globally. Instead of asking "How much can I prune this layer?", we should ask "Where in the entire model can I afford to lose information?"

Several recent frameworks address this:

  • Adaptive Layer Sparsity (ALS): Wei Li developed ALS to move away from uniform pruning. By assessing activation correlations across the whole model, it uses linear programming to identify "resilient" layers that can handle high sparsity, preserving capacity for "critical" layers.
  • Layer-Wise Error Model Search (LEMS): In Advancing SVD-based LLM Compression, Moritz Thoma uses an Integer Linear Program to find the optimal Singular Value Decomposition (SVD) rank for every layer simultaneously. This global trade-off allows for up to 80% compression by sacrificing precision in layers where the impact on final accuracy is minimal.
  • Genetic Algorithms: Mohammad Samragh proposed ASCAI, which treats the compression settings of all layers as a single optimization vector. This allows the search process to discover non-obvious synergies, such as where aggressive compression in Layer 5 is only viable if Layer 2 remains dense.

Holistic Efficiency: Energy and Hardware

The shift toward global optimization isn't just about accuracy; it's about real-world constraints like battery life and latency. Haichuan Yang points out in ECC that energy consumption is a system-level property. Optimizing layers independently fails to capture the cumulative data movement costs that drain a mobile device's battery.

By treating the network as a global graph, frameworks like EGRL can optimize how tensors are placed in memory across the entire execution pipeline. This global perspective identifies bottlenecks that a layer-wise compiler would miss, such as incompatible data layouts between two high-speed primitives that create a latency penalty when connected.

In summary, the field is moving from local heuristics to global search. By acknowledging that a neural network is more than the sum of its layers, researchers are unlocking higher compression ratios and faster inference speeds without the traditional sacrifice in model intelligence.

Papers in this version

Showing 60 of 89 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_global_optimization_of_compressed_neural_4cbb8848,
  title = {Global Optimization of Compressed Neural Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-optimization-of-compressed-neural-architectures/rdr_4cbb8848804d4b369f70e7824a92a3f8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_global_optimization_of_compressed_neural_4cbb8848_v1,
  title = {Global Optimization of Compressed Neural Architectures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/global-optimization-of-compressed-neural-architectures/rdr_4cbb8848804d4b369f70e7824a92a3f8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}