Joint Layer Optimization for Deep Network Compression

45 concepts46 papers

When compressing a deep neural network—whether through pruning, quantization, or low-rank decomposition—the standard playbook relies on a greedy, layer-by-layer approach. Each layer is treated as an isolated optimization problem: weights are removed or quantized to minimize that specific layer's local reconstruction error.

While computationally efficient, this localized strategy suffers from a fundamental flaw: it neglects inter-layer dependencies. A neural network is not a collection of independent blocks, but a highly coupled pipeline where the output distribution of one layer dictates the behavior of the next. Optimizing each layer in isolation causes approximation errors to compound non-linearly across the architecture, severely limiting the global compression rate and final accuracy.

To overcome this "layer-wise optimization ceiling," a prominent research direction focuses on joint layer optimization. By treating network compression as a holistic, end-to-end problem, these methods capture cross-layer interactions, allowing errors introduced in one part of the network to be compensated for or canceled out elsewhere.


The Sub-Optimality of Greedy Layer-Wise Compression

To understand why greedy layer-wise optimization fails, consider how information flows through a deep network. When a single layer is compressed, its output feature maps are altered. The subsequent layer, which was trained on the original, unperturbed feature distribution, must now process these distorted inputs. If that subsequent layer is also compressed using local metrics, it optimizes for an input distribution that no longer exists.

As noted by Yang et al. and Samragh et al., these local optimization strategies fail to account for complex inter-layer relationships, resulting in highly suboptimal global configurations. This sequential blind spot manifests in two primary ways:

  1. Misallocated Sparsity: Local metrics cannot determine if a layer's features are globally redundant or critical to the final task. As Kim et al. point out, an isolated approach overlooks opportunities to aggressively compress highly redundant layers while preserving the capacity of more sensitive, critical layers.
  2. Compounding Cascades: Small local errors accumulate into massive global distortions. Zhou argues that greedy layer-wise strategies inherently prioritize immediate local improvements at the expense of the global feature hierarchies discoverable only through simultaneous optimization.

Quantifying Inter-Layer Dependencies

To move beyond greedy heuristics, we must explicitly model how layers are coupled. Recent frameworks achieve this by analyzing structural, statistical, and routing dependencies across the network.

Structural Channel Coupling

In convolutional and transformer architectures, the output channels of layer $L$ directly form the input channels of layer $L+1$. Standard pruning methods often ignore this direct link, removing filters independently. To fix this, Li et al. introduce a regularization framework that treats the output channel of one layer and the corresponding input channel of the next as a single functional unit. Taking this further, Jeong et al. model network compression as a discrete Quadratically Constrained Quadratic Program (QCQP). By explicitly capturing the "quadratic coupling" between adjacent layers, their method synchronizes filter removal across functional dependencies, capturing redundancies that independent layer pruning misses entirely.

Activation and Routing Correlations

In complex architectures like Mixture-of-Experts (MoEs) or Large Language Models (LLMs), layers are functionally coupled through dynamic routing and activation distributions. For instance, Yang et al. demonstrate that pruning decisions made in isolation disrupt downstream routing logic and feature distributions in MoE models.

To preserve the end-to-end data flow, modern compression frameworks use global activation statistics as a proxy for cross-layer dependencies:

  • Activation Entropy: Wang et al. utilize activation entropy to capture and manage cross-layer dependencies during post-training quantization for vision-language models, preventing accuracy loss at low bit-widths.
  • Activation Correlations: Li et al. measure redundancy between layers via activation correlations, solving a global linear programming problem to intelligently allocate higher sparsity to redundant layers while protecting critical ones.

Mechanisms for Global Optimization

If local optimization is flawed, how can we optimize an entire network globally without encountering intractable computational complexity? Researchers have developed several elegant mechanisms to make global joint optimization practical.

Global Error Reconstruction

Instead of minimizing reconstruction error at the boundary of each layer, global methods minimize the discrepancy at the final output of the network. Lin et al. developed Global Error Reconstruction (GER), a framework that tracks how compression errors propagate through non-linear activations across the entire system. To scale this concept without prohibitive computational overhead, Lin et al. leverage Generative Adversarial Learning (GAL). They treat the compressed network as a generator tasked with mimicking the end-to-end feature outputs of the full baseline model, forcing the optimization to consider global feature interactions.

Additive Distortion and Dynamic Programming

A key breakthrough in making joint optimization efficient relies on discovering the underlying mathematical structure of pruning errors. Xu et al. discovered that the final output distortion caused by pruning individual layers is approximately additive. Because the joint impact can be broken down into an additive combination of layer-wise distortions, the authors successfully applied dynamic programming to find the mathematically optimal pruning ratio for every layer simultaneously in linear time.

Global Matrix and Tensor Formulations

Another strategy eliminates layer boundaries entirely during the compression phase by reformulating the network as a singular mathematical entity. Kim et al. developed Efficient Neural network Compression (ENC), which uses Singular Value Decomposition (SVD) to compute compression parameters across the entire network architecture at once.

In an even more radical formulation, Yin et al. treat an entire recurrent network as a single "mega-layer." By applying a Hierarchical Tucker tensor decomposition across the global network structure, they optimize all parameters simultaneously, capturing overarching structural correlations that disjointed, layer-by-layer optimization completely misses.

Optimization Strategy Core Mechanism How it Captures Inter-Layer Dependencies
Greedy Layer-Wise Local MSE minimization per layer Does not. Treats layers as isolated components; errors compound downstream.
Channel Coupling (QCQP) Discrete quadratic programming Models the explicit structural dependencies between output and input channels of adjacent layers.
Global Error Reconstruction Adversarial / End-to-end alignment Minimizes final output discrepancy, allowing downstream layers to adapt to upstream changes.
Additive Distortion (DP) Dynamic programming over error space Exploits the additive nature of layer distortions to compute globally optimal sparsity ratios in linear time.
Global Tensor Decomposition Mega-layer tensor factorization Eliminates layer boundaries entirely, optimizing the entire network structure as a single entity.

Key Foundational Work

Refining Architectures of Deep Convolutional Neural Networks (2016) — contributed 1 concept elements to this direction.
Towards Convolutional Neural Networks Compression via Global Error Reconstruction (2016) — contributed 1 concept elements to this direction.
Entropy-based Pruning for Phrase-based Machine Translation (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_greedy_layer_wise_optimization_8f0d0021,
  title = {greedy layer-wise optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/greedy-layer-wise-optimization/rdr_8f0d0021ce304e219a82053c0cd7d473},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_greedy_layer_wise_optimization_8f0d0021_v1,
  title = {greedy layer-wise optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/greedy-layer-wise-optimization/rdr_8f0d0021ce304e219a82053c0cd7d473/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake