Fusing Normalization Layers for Efficient Neural Network Inference

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2120 papers152 conceptsfrozen 2026-10-04

In the quest for faster neural networks, researchers have traditionally focused on reducing FLOPs (floating-point operations). However, as Jierun Chen argues in Run, Don't Walk: Chasing Higher FLOPS for Faster Neural Networks, theoretical operation counts often fail to predict real-world latency. A major culprit is the normalization layer. While essential for training stability, layers like Batch Normalization (BN) and Layer Normalization (LN) often become memory-bound bottlenecks during inference, requiring separate passes over data and high-bandwidth memory access.

The research direction emerging across these papers is a shift toward normalization fusion and structural re-parameterization. The goal is to treat normalization not as a distinct computational step, but as a mathematical property that can be "folded" directly into the weights of adjacent layers.

The Intuition of Folding

At its core, Batch Normalization is a linear transformation during inference. It scales and shifts activations using fixed statistics (mean and variance) learned during training. Because the preceding layer—usually a convolution or a linear projection—is also a linear transformation, the two can be mathematically collapsed into one.

As described in the foundational Google whitepaper by Raghuraman Krishnamoorthi, Quantizing deep convolutional networks for efficient inference, this "folding" process merges the normalization parameters directly into the weights and biases of the preceding convolutional layer. The result is a model that behaves identically but executes fewer operations and, more importantly, requires fewer memory reads.

From LayerNorm to BatchNorm

A significant friction point in modern architectures like Transformers is Layer Normalization. Unlike Batch Normalization, which uses static statistics at inference time, LayerNorm calculates the mean and variance for every input dynamically. This "online" calculation prevents simple folding.

To solve this, researchers like Peiyan Dong in HotBEV and Jialong Guo in SLAB propose replacing LayerNorm with BatchNorm in Transformer backbones. By doing so, they enable inference-time fusion that can reduce latency by up to 15% on GPUs. Qiming Yang takes this further with Unified Normalization, which uses "offline" normalization to eliminate the need for dedicated normalization kernels entirely, achieving a 31% throughput increase.

Structural Re-parameterization

The most sophisticated evolution of this idea is structural re-parameterization, popularized by Xiangyu Zhang and Xiaohan Ding in RepVGG: Making VGG-Style ConvNets Great Again.

The intuition here is to decouple the training architecture from the inference architecture:

  1. Training: Use a complex, multi-branch topology (e.g., $3 \times 3$ conv + $1 \times 1$ conv + identity) where each branch has its own Batch Normalization to ensure stable gradients.
  2. Inference: Mathematically collapse all those branches and their respective BN layers into a single $3 \times 3$ convolutional kernel.

This allows a model to benefit from the optimization advantages of complex architectures while running with the lightning-fast simplicity of a VGG-style stack. This principle is extended to Vision Transformers in RePaViT, where Xuwei Xu uses re-parameterization to compress feedforward networks, gaining both speed and accuracy.

Beyond Simple Layers

The concept of fusion is now reaching into specialized hardware and distributed systems:

  • Threshold Fusion: In binary neural networks, Arash Fayyazi shows in BlendNet that BN parameters can be fused directly into the activation thresholds, turning normalization into a simple comparison.
  • Communication Fusion: For distributed LLM inference, Raja Gond introduces TokenWeave, which fuses normalization into the communication kernels that synchronize data across GPUs, hiding the cost of normalization inside the unavoidable latency of networking.
  • Lookup Tables: Longguang Wang demonstrates in the Deep Lookup Network that BN can be pre-calculated and baked into the entries of a lookup table, enabling multiplication-free inference.

Why It Works

Normalization fusion works because it resolves the tension between optimization and execution. Training requires normalization to smooth the loss landscape and prevent internal covariate shift. Execution, however, rewards high arithmetic intensity and minimal memory movement. By treating normalization as a re-parameterization problem, researchers allow the model to "forget" its training-time complexity and settle into a hardware-optimized form for deployment.

Papers in this version

Showing 60 of 120 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 100 papers
  2. Version 1 · 2026-10-04 · 120 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_batch_normalization_folding_336d307d,
  title = {batch normalization folding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/batch-normalization-folding/rdr_336d307d168b4bf6a85a67d144ef71ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_fusing_normalization_layers_for_efficien_336d307d_v1,
  title = {Fusing Normalization Layers for Efficient Neural Network Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/fusing-normalization-layers-for-efficient-neural-network-inference/rdr_336d307d168b4bf6a85a67d144ef71ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}