Class-Balanced Normalization for Imbalanced Data

22 concepts11 papers1 proposal

In the idealized world of machine learning research, data is often independent and identically distributed (i.e., i.i.d.) and perfectly balanced across classes. In reality, deep networks encounter highly skewed distributions. Whether training on a long-tailed dataset where a few "head" classes dominate the majority of samples, or deploying a model to a real-world data stream that encounters environments in highly correlated bursts, data imbalance is a pervasive challenge.

While many techniques attempt to address this issue by reweighting loss functions or adjusting the final classifier head, a deeper structural vulnerability exists inside the network itself: the normalization layers. Standard normalization techniques—such as Batch Normalization (BN) and Layer Normalization (LN)—inadvertently encode class imbalance into the network's internal feature representations, suppressing minority classes and biasing predictions toward the majority.

To fix this, a growing body of research focuses on class-balanced normalization, restructuring how internal statistics are computed to ensure equitable feature representation.


The Hidden Assumption in Batch Statistics

To understand why standard normalization fails, we must look at how it calculates statistics. Batch Normalization stabilizes training by computing a empirical mean ($\mu$) and variance ($\sigma^2$) across all samples in a mini-batch:

$$\mu = \frac{1}{N}\sum_{i=1}^N x_i$$

This formulation relies on a critical, hidden assumption: that every sample $x_i$ in the batch should contribute equally to the global distribution estimate.

When a batch is heavily imbalanced—for instance, containing 95% samples of "cats" and 5% of "dogs"—this assumption breaks down. As Sang-Hoon Lee et al. observe in OIMNet++: Prototypical Normalization and Localization-Aware Learning for Person Search, frequent identities completely dominate the mean and variance calculations. The resulting normalization standardizes the entire batch relative to the majority class's features. Consequently, the unique, discriminative features of minority classes are treated as outliers and effectively erased or distorted during the scaling process.


How Imbalance Corrupts the Feature Space

This statistical skew corrupts the network's internal representations in two primary ways:

1. Single-Gaussian Distortion

Standard BN models feature statistics under the assumption of a single Gaussian distribution. Lechao Cheng et al. point out in Compound Batch Normalization for Long-tailed Image Classification that this single-component assumption fails in long-tailed scenarios because the distribution is heavily pulled toward head classes, causing a detrimental internal covariate shift for tail classes. Over multiple iterations, these skewed statistics accumulate. As noted by Xiao Cui et al. in Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling, the running averages stored by the model become corrupted distributional anchors that misrepresent the true diversity of the dataset.

2. Normalization Parameter Drift

The issue extends beyond the running statistics to the learnable affine parameters (the scaling factor $\gamma$ and bias $\beta$). Guoliang Kang et al. demonstrate in Normalizing Batch Normalization for Long-Tailed Recognition that when these parameters are unconstrained, the network allows them to actively suppress the magnitude of features associated with rare classes, rendering them less discriminative.

This vulnerability is especially severe during inference. The authors of Logit-Adjusted Test-Time Adaptation under Partial Class Imbalance identify a phenomenon called beta-drift: when a model encounters a test stream containing only a subset of classes, the learnable bias parameter ($\beta$) drifts rapidly toward the dominant classes present in the test data, locking the model into biased predictions. This susceptibility of normalization parameters to noise and task imbalance is a shared weakness across architectures, also appearing in Layer Normalization layers within multilingual models (Xian Li et al. in Robust Optimization for Multilingual Translation with Imbalanced Data).


Restoring Balance: Class-Aware and Prototypical Statistics

To prevent majority classes from overriding the feature space, recent methods decouple normalization statistics from raw sample counts. The core intuition is simple: give every semantic category an equal vote in the normalization process, regardless of how many samples it has.

Standard Batch Normalization (Sample-Level):
[ Cat, Cat, Cat, Cat, Cat, Dog ] ---> Mean/Var dominated by "Cat"

Class-Balanced Normalization (Class-Level):
[ Cat, Cat, Cat, Cat, Cat ] ---> Compute Cat Prototype \
                                                         ---> Average Mean/Var equally
[ Dog                     ] ---> Compute Dog Prototype /

Prototypical Normalization

Instead of computing statistics across individual batch samples, Sang-Hoon Lee et al. introduce ProtoNorm. This mechanism first aggregates samples into class-averaged representations (prototypes) and then calculates the mean and variance from these prototypes. Because each class yields exactly one prototype, minority classes contribute equally to the feature scaling process.

Class-Aware Averaging via Pseudo-Labels

When true labels are unavailable—such as during test-time adaptation—models must rely on online estimation. Yongyi Su et al. develop a framework in Towards Real-World Test-Time Adaptation: Tri-net Self-Training with Balanced Normalization that uses model pseudo-labels to maintain separate, category-specific mean and variance statistics. Similarly, Chaoqun Du et al. implement this in UniTTA: Unified Benchmark and Versatile Framework Towards Realistic Test-Time Adaptation by computing normalization statistics independently for each predicted class within a domain and then averaging them. This prevents majority classes from suppressing minority features.

Furthermore, these specialized layers can be designed to accommodate structural differences between classes. For instance, tail-class features often exhibit higher statistical variance in the latent space than head-class features. Chengcheng Ma et al. demonstrate in Three Heads Are Better than One: Complementary Experts for Long-Tailed Semi-supervised Learning that explicitly accounting for these class-wise variance discrepancies stabilizes training and yields more consistent latent representations.


The Critical Frontier: Test-Time Adaptation

The necessity of class-balanced normalization becomes acutely apparent in Test-Time Adaptation (TTA), where a pre-trained model must adapt to a shifting deployment environment on the fly. Real-world data streams are rarely i.i.d.; they are highly temporally correlated.

As Taesik Gong et al. illustrate in Robust Continual Test-time Adaptation Against Temporal Correlation, a self-driving car traveling through a long tunnel will process minutes of highly correlated, non-random data (e.g., concrete walls). If standard TTA methods blindly update batch normalization statistics on these skewed streams, the model's internal distribution estimates collapse.

W. Ma demonstrates in Dynamic Adaptation for Class-Imbalanced Streams: An Imbalanced Continuous Test-Time Framework that standard adaptation techniques like entropy minimization break down under these conditions because they lack mechanisms to handle unequal class representation, causing the model to completely ignore minority classes. By shifting to instance-level normalization or tracking class-balanced statistics dynamically, models can successfully adapt to environmental shifts (like weather changes) without allowing temporary class imbalances to corrupt their underlying feature space.

Key Foundational Work

OIMNet++: Prototypical Normalization and Localization-Aware Learning for Person Search (2022) — contributed 3 concept elements to this direction.
Compound Batch Normalization for Long-tailed Image Classification (2022) — contributed 2 concept elements to this direction.
NOTE: Robust Continual Test-time Adaptation Against Temporal Correlation (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_class_balanced_normalization_34192b52,
  title = {class-balanced normalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-balanced-normalization/rdr_34192b524b054217b31f8d0908f5a6e7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_class_balanced_normalization_34192b52_v1,
  title = {class-balanced normalization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-balanced-normalization/rdr_34192b524b054217b31f8d0908f5a6e7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake