Normalization layers, such as Batch Normalization (BN) and Layer Normalization (LN), are indispensable tools in modern deep learning. By stabilizing internal covariate shift and smoothing the loss landscape, they enable deep networks to converge reliably and quickly.
However, when deploying a trained model to production environments—such as mobile GPUs, edge NPUs, or custom hardware accelerators—these normalization layers become a computational liability. Executing them as distinct steps introduces separate element-wise operations, forces memory round-trips, and requires hardware-unfriendly math like square roots. In resource-constrained settings, these non-computational layers can dominate execution latency, as noted by Seock-Hwan Noh et al. in LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training.
The key insight that resolves this tension is normalization folding (or fusion). During inference, the statistics used by a normalization layer are no longer dynamic; they are frozen running constants. Because both the preceding linear layer (such as a convolution or fully connected layer) and the normalization layer are affine transformations, they can be mathematically fused into a single linear operation. At runtime, the normalization layer completely disappears.
The Mathematics of Fusion
To understand why normalization folding works, we can look at the underlying linear algebra. Consider a standard convolutional or fully connected layer that takes an input $X$ and computes an intermediate output $Y$ using a weight matrix $W$ and a bias vector $B$:
$$Y = W \cdot X + B$$
In a typical network architecture, a Batch Normalization layer immediately follows. During inference, it scales and shifts $Y$ using the learned parameters ($\gamma, \beta$) and the frozen running statistics ($\mu, \sigma^2$):
$$Z = \gamma \cdot \frac{Y - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$
Because $Y$ is simply $W \cdot X + B$, we can substitute the first equation directly into the second:
$$Z = \gamma \cdot \frac{(W \cdot X + B) - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$
By distributing the denominator and grouping the terms that multiply the input $X$, we can rearrange this equation into a standard affine form ($Z = W' \cdot X + B'$):
$$Z = \left( \frac{\gamma \cdot W}{\sqrt{\sigma^2 + \epsilon}} \right) \cdot X + \left( \gamma \cdot \frac{B - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \right)$$
Because $\gamma, \mu, \sigma^2, \epsilon, B,$ and $W$ are all static constants during deployment, we can pre-compute a new, folded weight matrix $W'$ and a folded bias vector $B'$ before the model is compiled:
$$W' = \frac{\gamma \cdot W}{\sqrt{\sigma^2 + \epsilon}}$$
$$B' = \gamma \cdot \frac{B - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$
By replacing the original weights and biases with $W'$ and $B'$, the normalization layer is entirely absorbed. As demonstrated by Edouard Yvinec et al. in To Fold or Not to Fold: a Necessary and Sufficient Condition on Batch-Normalization Layers Folding, this transformation maintains exact mathematical equivalence with the original network. It delivers significant inference speedups across hardware platforms with zero loss in accuracy.
This closed-form integration underpins structural re-parameterization techniques, such as the VGG-style conversion proposed by Xiangyu Zhang in RepVGG: Making VGG-Style ConvNets Great Again, and the layer-slimming methods detailed in DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices by Dawei Li 0006 et al..
The LayerNorm Bottleneck and the Shift to BatchNorm
While folding is straightforward for Batch Normalization, modern architectures like Transformers rely heavily on Layer Normalization (LN). Unlike BatchNorm, LayerNorm computes its mean and variance dynamically for every single input token at runtime. This dependency prevents static folding because $\mu$ and $\sigma^2$ change with every forward pass.
To circumvent this bottleneck and unlock inference-time fusion in Transformer models, researchers employ two primary strategies:
- Replacing LayerNorm with BatchNorm: By structurally swapping LN for BN during training, the model retains optimization stability while allowing the normalization parameters to be fused into linear layers at inference time. For instance, Jialong Guo et al. introduced SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization, demonstrating that this swap eliminates the need for real-time statistic calculations, significantly reducing latency on GPUs and NPUs. Similarly, Peiyan Dong et al. utilized this substitution in HotBEV: Hardware-oriented Transformer-based Multi-View 3D Detector for BEV Perception to reduce memory access overhead. Single-author works like Xuwei Xu's RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers and Chuanyang Zheng's IFORMER: INTEGRATING CONVNET AND TRANSFORMER FOR MOBILE APPLICATION also leverage this design to fold normalization layers into adjacent linear weights for mobile deployment.
- Static Approximation of LayerNorm: Alternatively, researchers have found ways to convert dynamic normalization into static scaling factors. In You can remove GPT2's LayerNorm by fine-tuning, Stefan Heimersheim demonstrated that by calculating the average standard deviation of activations across a large dataset during a fine-tuning phase, the non-linear LayerNorm operation can be replaced with a static scaling factor and successfully folded into adjacent weight matrices.
Co-Design with Quantization and Specialized Hardware
Normalization folding is rarely an isolated optimization; it serves as a critical pre-processing step for low-precision quantization and hardware co-design.
When quantizing a network to integer-only arithmetic (such as INT8 or INT4), keeping normalization layers separate forces the hardware to drop out of integer logic to compute floating-point scales and square roots. As established by Benoit Jacob et al. in Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, folding batch normalization parameters directly into weights before quantization eliminates this runtime penalty entirely.
This synergy is heavily exploited in edge deployments, such as the ultra-low-power brain-machine interface developed by Xiaying Wang in Q-EEGNet: an Energy-Efficient 8-bit Quantized Parallel EEGNet Implementation for Edge Motor-Imagery Brain-Machine Interfaces, and the autonomous navigation engine designed by Luca Benini in An Open Source and Open Hardware Deep Learning-powered Visual Navigation Engine for Autonomous Nano-UAVs.
The principle of folding also extends naturally to non-traditional computing paradigms:
- Spiking Neural Networks (SNNs): Instead of folding into weights, Yufei Guo et al. demonstrated in Membrane Potential Batch Normalization for Spiking Neural Networks that element-wise normalization parameters can be folded directly into the firing thresholds of spiking neurons, providing fine-grained control with zero inference overhead.
- Lookup Table Networks: In architectures that replace multiplications with lookups, such as Deep Lookup Network by Longguang Wang et al., normalization and scaling factors are integrated directly into the lookup table entries themselves.
- Analog and Neuromorphic Hardware: In hardware-software co-designs like those explored by Gourav Datta et al. in In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Vision, normalization scale factors are mapped directly to physical transistor widths and shift factors to the threshold levels of analog comparators.
Ultimately, normalization folding works because it respects the mathematical structure of the network while adapting it to the realities of hardware execution. It allows practitioners to train models with the stability of normalization and deploy them with the speed of a pure, streamlined linear pipeline.
Key Foundational Work
Quantizing deep convolutional networks for efficient inference: A whitepaper (2018) — contributed 1 concept elements to this direction.