Deep neural networks derive their expressive power from the interleaving of linear transformations and non-linear activation functions. However, this depth comes at a cost: each layer adds computational latency, memory overhead, and sequential dependencies that modern hardware cannot always parallelize away.
A growing body of research suggests that much of this depth is redundant. By identifying layers that behave linearly or by forcing them to become linear during training, researchers can mathematically "collapse" multiple layers into one. This process, often called layer fusion or algebraic collapsing, allows a model to retain the optimization benefits of a deep architecture during training while operating with the speed of a shallow network during deployment.
The Core Intuition: Linear Composition
The mathematical foundation of this idea is simple: the composition of two linear transformations is itself a single linear transformation. If we have two consecutive weight matrices, $W_1$ and $W_2$, without a non-linear activation function between them, their operation on an input $x$ is: $$y = W_2(W_1 x) = (W_2 \cdot W_1)x$$ By pre-computing the product $W_{merged} = W_2 \cdot W_1$, we can replace two layers with one. As Haichuan Yang and Meng Li demonstrate in DepthShrinker, this transformation reduces the number of kernel launches on GPUs and TPUs, directly boosting hardware efficiency.
Identifying Redundant Non-Linearity
The barrier to collapsing layers is the non-linear activation function (like ReLU). To collapse a network's depth, one must first identify which activations are "lazy" or redundant.
Zhu Liao, Victor Quétu, and Enzo Tartaglione have pioneered methods to measure this redundancy using activation entropy. In their work on Entropy Guided Pruning and the EASIER framework, they show that if a layer's neurons operate almost exclusively in their linear region across the input distribution, the activation function can be replaced with an identity function. Once linearized, the layer can be folded into its neighbor.
Similarly, Silas Dobler proposes "linearity-based compression," which targets neurons that are always active (positive for ReLU). These neurons effectively act as linear shifts and can be algebraically merged into subsequent layers, reducing the model's parameter count without the accuracy drops typical of binary neuron removal.
Training for Collapse
Rather than just pruning existing models, some researchers design architectures specifically to be collapsed. This "expand-then-contract" strategy leverages the fact that over-parameterized, deeper models are often easier to train.
- ExpandNets: Shuxuan Guo and Mathieu Salzmann introduced ExpandNets, which replaces a single convolutional layer with a sequence of linear layers during training. After convergence, these are collapsed back into a single layer, providing the performance of a deep model at the cost of a shallow one.
- Structural Re-parameterization: In the context of wakeword detection, Arnav Kundu uses RepCNN to train multi-branched architectures that are mathematically summed into a single-branch model for inference, achieving 10x speedups.
- Transformable Networks: Xiangzhong Luo explores DW-NAS, searching for "deep-to-shallow transformable" architectures that optimize for this specific collapse potential.
Applications Beyond Speed
While inference speed is the primary driver, layer collapse solves critical problems in specialized domains:
- Private Inference (PI): In encrypted computation, non-linear operations like ReLU are orders of magnitude more expensive than linear ones. Nandan Kumar Jha and Siddharth Garg developed DeepReDuce to strategically drop ReLUs, enabling layer fusion that slashes the communication overhead of secure protocols.
- Model Obfuscation: To protect proprietary architectures from reverse engineering, Hui Xu uses DeepObfuscation to "flatten" complex structures like Inception blocks into opaque, sequential linear layers that hide the original design logic.
- Security: Soheil Zibakhsh Shabgahi utilizes layer merging in MergeGuard to thwart Trojan attacks, as the process of linearization and fusion can "wash out" the specific non-linear triggers used by attackers.
By treating network depth as a fluid property that can be expanded for learning and collapsed for execution, this research direction provides a path toward models that are both highly expressive and hardware-efficient.