Reducing Network Depth through Linear Layer Folding

109 concepts68 papers2 proposals

Deep neural networks derive their remarkable expressive power from non-linear activation functions (such as ReLU or GeLU) interspersed between linear transformations (such as matrix multiplications or convolutions). Without these non-linearities, a deep network collapses mathematically into a shallow one.

Consider two consecutive linear layers without an intervening activation function:

$$y = W_2 (W_1 x)$$

Because matrix multiplication is associative, this sequence can be rewritten as a single operation:

$$y = (W_2 W_1) x = W_{\text{fused}} x$$

By pre-computing $W_{\text{fused}} = W_2 W_1$, we can replace two sequential matrix multiplications with one. This fundamental algebraic property forms the basis of linear layer folding (also referred to as layer fusion or collapsing).

Layer folding allows researchers to decouple the training architecture from the deployment architecture. We can design deep, highly parameterized networks that optimize efficiently, and then mathematically collapse them into compact, low-latency structures for real-world hardware deployment.


Training-Time Overparameterization

Why intentionally add linear layers during training if they are collapsed prior to deployment? The answer lies in optimization dynamics. Deep linear networks possess different gradient properties and optimization landscapes compared to shallow ones, acting as an implicit regularizer.

For instance, Li Jing 0001 et al. introduced the Implicit Rank-Minimizing Autoencoder (IRMAE), demonstrating that inserting extra linear layers into an autoencoder implicitly minimizes the rank of the learned representations via gradient descent. During inference, these auxiliary layers are collapsed into a single matrix, providing the benefits of low-rank regularization with zero added computational cost.

In computer vision, this strategy helps models learn complex spatial features without sacrificing inference speed. Kartikeya Bhardwaj et al. proposed Collapsible Linear Blocks for Super-Efficient Super Resolution (SESR), replacing single convolutional layers with sequences of linear convolutions during training to expand representational capacity. Post-training, these sequences are mathematically collapsed into a single narrow convolution, enabling real-time 4K upscaling on mobile hardware.

This approach is expanded by P. Soma et al. in Enhanced collapsible linear blocks for arbitrary sized image super-resolution, showing that linear overparameterization consistently improves performance across varying image scales without increasing deployment overhead. Similarly, the NetBooster framework introduced by Zhongzhi Yu et al. in NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giants leverages an "expansion-then-contraction" strategy to mitigate under-fitting in tiny neural networks, folding weights to achieve accuracy gains at zero inference cost.


Post-Training Linearization and Compression

Instead of designing collapsible blocks from scratch, another facet of this research focuses on identifying and removing redundant non-linearities in already-trained networks.

As Christian H. X. Ali Mehmeti-Göpel demonstrates in Nonlinear Advantage: Trained Networks Might Not Be As Complex as You Think, a vast majority of a trained network's non-linear complexity can be removed post-training with minimal performance loss. When neurons operate entirely within their linear regions across a dataset, their activation functions act as simple identity mappings.

To exploit this hidden linearity, Haichuan Yang et al. developed DepthShrinker: A New Compression Paradigm Towards Boosting Real-Hardware Efficiency of Compact Neural Networks. The framework identifies non-linearities that become redundant post-training, removes them, and collapses consecutive linear operations—such as pointwise and depthwise convolutions—into single dense convolutions that better exploit parallel processing hardware.

Training/Standard State:
[ Linear Layer 1 ] ---> [ Non-Linear Activation (Identity/Lazy) ] ---> [ Linear Layer 2 ]

Post-Training Linearization:
[ Linear Layer 1 ] --------------------------------------------------> [ Linear Layer 2 ]

Mathematical Fusion (Deployment):
[                    Fused Dense Linear Layer                    ]

To systematically isolate these target activations, Victor Quétu et al. introduced an entropy-based metric in The Simpler The Better: An Entropy-Based Importance Metric to Reduce Neural Networks' Depth, replacing "lazy" low-entropy non-linear activations with identity functions to enable operator folding.

Furthermore, Enzo Tartaglione et al. showed in NEPENTHE: Entropy-Based Pruning as a Neural Network Depth's Reducer that standard magnitude-based unstructured pruning naturally drives neuron activation entropy toward zero. This bridges the gap between unstructured weight pruning and structured depth reduction: as pruning linearizes layers, they can be mathematically fused into adjacent layers, yielding direct hardware latency reductions.


Overcoming Bottlenecks in Secure Inference and Verification

Linear layer folding is also a vital tool for specialized domains like Secure Multi-Party Computation (MPC), Homomorphic Encryption (HE), and formal verification, where non-linear functions introduce severe performance penalties.

In private inference, evaluating non-linear functions like ReLU or GeLU requires expensive cryptographic protocols and communication-heavy truncation steps. Linear operations, by contrast, are highly efficient. By selectively linearizing activations, researchers can drastically reduce cryptographic overhead.

Wenxuan Zeng et al. demonstrated in CoPriv: Network/Protocol Co-Optimization for Communication-Efficient Private Inference that removing non-linear activations allows adjacent linear layers to be structurally merged, eliminating intermediate fixed-point truncation steps and reducing online communication costs. Similarly, Hongwu Peng utilized structural linearization in LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference to ensure uniform multiplication depth and minimize cryptographic latency.

For formal verification, deep networks present a massive search space for Satisfiability Modulo Theories (SMT) solvers because every non-linear activation introduces new constraints and branching paths. As shown by Guy Amir et al. in An SMT-Based Approach for Verifying Binarized Neural Networks, merging consecutive linear operations (such as weighted sums and batch normalization) during a preprocessing phase reduces the total number of variables and constraints, simplifying the underlying optimization problem for verification tools.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 73 papers
  2. Version 1 · 2026-10-04 · 60 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_linear_layer_fusion_e6c2eef2,
  title = {linear layer fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-layer-fusion/rdr_e6c2eef2bee245b985f402d3e0a4c338},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_linear_layer_fusion_e6c2eef2_v2,
  title = {linear layer fusion},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-layer-fusion/rdr_e6c2eef2bee245b985f402d3e0a4c338/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake