Balanced Normalization for Imbalanced and Incremental Data

This direction is split into other directions: class-balanced normalization. Its last version stays readable and citable here.

Version 1 of 122 papers41 conceptsfrozen 2026-10-04

In the controlled environment of a research lab, data is usually "i.i.d."—independent and identically distributed. But in the messy reality of a self-driving car or a streaming health monitor, data is often temporally correlated or long-tailed. If a car drives through a tunnel, it sees concrete walls for minutes; if a model learns new tasks over time, it sees only the new classes while "forgetting" the old ones.

Standard Batch Normalization (BN), a cornerstone of modern deep learning, inadvertently becomes a bottleneck in these scenarios. Because BN calculates mean and variance across a mini-batch to scale features, it assumes that the batch is a representative micro-sample of the whole world. When data is imbalanced or shifting, this assumption fails, leading to "feature pollution" where majority classes or recent tasks drown out everything else.

The BN Dilemma: Why Global Statistics Fail

The core problem is a "majority rules" bias. In a long-tailed dataset, the "head" classes (the most frequent ones) dominate the feature statistics. As Lechao Cheng and colleagues observe in Compound Batch Normalization for Long-tailed Image Classification, standard BN models the feature space as a single Gaussian distribution. If 90% of your data belongs to one class, the normalization parameters will be tuned almost exclusively to that class, leaving minority "tail" classes poorly represented.

This issue is even more acute in Class-Incremental Learning (CIL). When a model learns Task B after Task A, the mini-batches are heavily skewed toward Task B. Sungmin Cha and Taesup Moon identify this as a primary trigger for "catastrophic forgetting." The running statistics of the BN layers quickly overwrite the knowledge of Task A to accommodate the new distribution of Task B.

Rebalancing via Class-Aware Statistics

To fix this, researchers have moved toward Class-Balanced Normalization. The intuition is simple: if the batch is imbalanced, don't calculate statistics across the whole batch. Instead, calculate them for each class individually and then combine them.

In the context of Test-Time Adaptation (TTA), Yongyi Su and Xun Xu proposed TRIBE, which uses pseudo-labels to track separate mean and variance statistics for every semantic category. By averaging these class-specific statistics into a global estimate, the model ensures that minority classes are not "washed out" during on-the-fly updates. A similar approach is taken by Amin Ahsan Ali in pFedBBN, where class-aware normalization prevents performance collapse in federated learning settings where different users have wildly different class distributions.

Virtual Batches and Prototypical Anchors

When labels aren't available or batches are too small to have representatives of every class, researchers use "virtual" rebalancing or "prototypes."

  1. Virtual Batch Rebalancing: In Task-Balanced Batch Normalization (TBBN), Sungmin Cha introduces a "reshape and repeat" strategy. By repeating feature maps from past tasks (stored in a small buffer) and reshaping current task features, they create a synthetic, balanced mini-batch. This allows standard BN layers to compute gradients that reflect an equal distribution between old and new data without actually changing the training set.
  2. Prototypical Normalization: Sang-Hoon Lee and colleagues developed ProtoNorm for person search. Instead of using raw samples, they calculate normalization statistics from class-level prototypes (averaged representations of an identity). This ensures that a person who appears in 100 frames has the same influence on the model's scaling as a person who appears in only two.

Handling Statistical Variance in the Tail

A subtle but critical finding by Chengcheng Ma and Ismail Elezi is that tail classes don't just have fewer samples—they often have higher feature variance.

In Complementary Experts for Long-Tailed Semi-supervised Learning, they argue that standard BN fails because it tries to force these high-variance tail features into the same statistical mold as low-variance head features. Their solution, Classwise Batch Normalization (CBN), partitions features into groups based on frequency (e.g., "head," "medium," and "tail") and applies separate normalization layers to each. This prevents the noisy, high-variance representations of rare classes from "polluting" the stable representations of common ones.

Decoupling Statistics from Weights

Finally, some researchers argue that the best way to handle shifting data is to stop updating BN statistics during the main training pass. Minghao Zhou and Deyu Meng suggest a "BN decoupling" approach in Diagnosing Batch Normalization in Class Incremental Learning.

Instead of letting the biased stream of new data update the running mean and variance, they perform a dedicated, post-hoc forward pass using a balanced memory buffer. This ensures that the final statistics used for inference are a fair reflection of the entire data distribution the model has seen, rather than a snapshot of the most recent (and likely biased) task. Similarly, Quanziang Wang uses this re-estimation technique in Dual-CBA to bridge the "stability gap" in online continual learning.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_balanced_normalization_for_imbalanced_an_ac3866dd,
  title = {Balanced Normalization for Imbalanced and Incremental Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/balanced-normalization-for-imbalanced-and-incremental-data/rdr_ac3866dd1e344902b75294014b72d59a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_balanced_normalization_for_imbalanced_an_ac3866dd_v1,
  title = {Balanced Normalization for Imbalanced and Incremental Data},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/balanced-normalization-for-imbalanced-and-incremental-data/rdr_ac3866dd1e344902b75294014b72d59a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}