Folding Normalization Layers for Efficient Inference

141 concepts96 papers1 proposal

Normalization layers, such as Batch Normalization (BN) and Layer Normalization (LN), are indispensable tools in modern deep learning. By stabilizing internal covariate shift and smoothing the loss landscape, they enable deep networks to converge reliably and quickly.

However, when deploying a trained model to production environments—such as mobile GPUs, edge NPUs, or custom hardware accelerators—these normalization layers become a computational liability. Executing them as distinct steps introduces separate element-wise operations, forces memory round-trips, and requires hardware-unfriendly math like square roots. In resource-constrained settings, these non-computational layers can dominate execution latency, as noted by Seock-Hwan Noh et al. in LightNorm: Area and Energy-Efficient Batch Normalization Hardware for On-Device DNN Training.

The key insight that resolves this tension is normalization folding (or fusion). During inference, the statistics used by a normalization layer are no longer dynamic; they are frozen running constants. Because both the preceding linear layer (such as a convolution or fully connected layer) and the normalization layer are affine transformations, they can be mathematically fused into a single linear operation. At runtime, the normalization layer completely disappears.

The Mathematics of Fusion

To understand why normalization folding works, we can look at the underlying linear algebra. Consider a standard convolutional or fully connected layer that takes an input $X$ and computes an intermediate output $Y$ using a weight matrix $W$ and a bias vector $B$:

$$Y = W \cdot X + B$$

In a typical network architecture, a Batch Normalization layer immediately follows. During inference, it scales and shifts $Y$ using the learned parameters ($\gamma, \beta$) and the frozen running statistics ($\mu, \sigma^2$):

$$Z = \gamma \cdot \frac{Y - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

Because $Y$ is simply $W \cdot X + B$, we can substitute the first equation directly into the second:

$$Z = \gamma \cdot \frac{(W \cdot X + B) - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

By distributing the denominator and grouping the terms that multiply the input $X$, we can rearrange this equation into a standard affine form ($Z = W' \cdot X + B'$):

$$Z = \left( \frac{\gamma \cdot W}{\sqrt{\sigma^2 + \epsilon}} \right) \cdot X + \left( \gamma \cdot \frac{B - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \right)$$

Because $\gamma, \mu, \sigma^2, \epsilon, B,$ and $W$ are all static constants during deployment, we can pre-compute a new, folded weight matrix $W'$ and a folded bias vector $B'$ before the model is compiled:

$$W' = \frac{\gamma \cdot W}{\sqrt{\sigma^2 + \epsilon}}$$

$$B' = \gamma \cdot \frac{B - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta$$

By replacing the original weights and biases with $W'$ and $B'$, the normalization layer is entirely absorbed. As demonstrated by Edouard Yvinec et al. in To Fold or Not to Fold: a Necessary and Sufficient Condition on Batch-Normalization Layers Folding, this transformation maintains exact mathematical equivalence with the original network. It delivers significant inference speedups across hardware platforms with zero loss in accuracy.

This closed-form integration underpins structural re-parameterization techniques, such as the VGG-style conversion proposed by Xiangyu Zhang in RepVGG: Making VGG-Style ConvNets Great Again, and the layer-slimming methods detailed in DeepRebirth: Accelerating Deep Neural Network Execution on Mobile Devices by Dawei Li 0006 et al..

The LayerNorm Bottleneck and the Shift to BatchNorm

While folding is straightforward for Batch Normalization, modern architectures like Transformers rely heavily on Layer Normalization (LN). Unlike BatchNorm, LayerNorm computes its mean and variance dynamically for every single input token at runtime. This dependency prevents static folding because $\mu$ and $\sigma^2$ change with every forward pass.

To circumvent this bottleneck and unlock inference-time fusion in Transformer models, researchers employ two primary strategies:

  1. Replacing LayerNorm with BatchNorm: By structurally swapping LN for BN during training, the model retains optimization stability while allowing the normalization parameters to be fused into linear layers at inference time. For instance, Jialong Guo et al. introduced SLAB: Efficient Transformers with Simplified Linear Attention and Progressive Re-parameterized Batch Normalization, demonstrating that this swap eliminates the need for real-time statistic calculations, significantly reducing latency on GPUs and NPUs. Similarly, Peiyan Dong et al. utilized this substitution in HotBEV: Hardware-oriented Transformer-based Multi-View 3D Detector for BEV Perception to reduce memory access overhead. Single-author works like Xuwei Xu's RePaViT: Scalable Vision Transformer Acceleration via Structural Reparameterization on Feedforward Network Layers and Chuanyang Zheng's IFORMER: INTEGRATING CONVNET AND TRANSFORMER FOR MOBILE APPLICATION also leverage this design to fold normalization layers into adjacent linear weights for mobile deployment.
  2. Static Approximation of LayerNorm: Alternatively, researchers have found ways to convert dynamic normalization into static scaling factors. In You can remove GPT2's LayerNorm by fine-tuning, Stefan Heimersheim demonstrated that by calculating the average standard deviation of activations across a large dataset during a fine-tuning phase, the non-linear LayerNorm operation can be replaced with a static scaling factor and successfully folded into adjacent weight matrices.

Co-Design with Quantization and Specialized Hardware

Normalization folding is rarely an isolated optimization; it serves as a critical pre-processing step for low-precision quantization and hardware co-design.

When quantizing a network to integer-only arithmetic (such as INT8 or INT4), keeping normalization layers separate forces the hardware to drop out of integer logic to compute floating-point scales and square roots. As established by Benoit Jacob et al. in Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference, folding batch normalization parameters directly into weights before quantization eliminates this runtime penalty entirely.

This synergy is heavily exploited in edge deployments, such as the ultra-low-power brain-machine interface developed by Xiaying Wang in Q-EEGNet: an Energy-Efficient 8-bit Quantized Parallel EEGNet Implementation for Edge Motor-Imagery Brain-Machine Interfaces, and the autonomous navigation engine designed by Luca Benini in An Open Source and Open Hardware Deep Learning-powered Visual Navigation Engine for Autonomous Nano-UAVs.

The principle of folding also extends naturally to non-traditional computing paradigms:

Ultimately, normalization folding works because it respects the mathematical structure of the network while adapting it to the realities of hardware execution. It allows practitioners to train models with the stability of normalization and deploy them with the speed of a pure, streamlined linear pipeline.

Key Foundational Work

Quantizing deep convolutional networks for efficient inference: A whitepaper (2018) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 100 papers
  2. Version 1 · 2026-10-04 · 120 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_batch_normalization_folding_336d307d,
  title = {batch normalization folding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/batch-normalization-folding/rdr_336d307d168b4bf6a85a67d144ef71ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_batch_normalization_folding_336d307d_v2,
  title = {batch normalization folding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/batch-normalization-folding/rdr_336d307d168b4bf6a85a67d144ef71ba/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake