Dynamic Class Distribution Estimation for Label Shift

56 concepts47 papers1 proposal

When we deploy machine learning models into the real world, we often assume that the environment will mirror our training data. In practice, however, class frequencies shift constantly. A spam filter might encounter a sudden surge of phishing emails during a holiday; a medical diagnostic tool might face an unexpected outbreak of a rare disease. This phenomenon is known as label shift (or class distribution mismatch).

Traditional classifiers assume a static, often uniform, prior distribution over classes. When confronted with a skewed test stream, they suffer from prediction collapse, over-predicting historical majority classes and ignoring rare ones. Early foundational work by Lipton et al. demonstrated that we can correct for label shift by applying importance weights to a model's outputs, provided we can accurately estimate the new target-domain class priors.

But how do we estimate these priors when we are receiving unlabeled data in real-time? The solution lies in Dynamic Class Distribution Estimation: tracking and updating the class prior on the fly by leveraging the model's own evolving predictions.


Tracking Priors via Running Statistics

The core intuition behind dynamic estimation is that a model's sequential predictions, while individually imperfect, contain an aggregate signal about the true underlying distribution of the data stream. Instead of waiting for an entire dataset to arrive to calculate global statistics, models can maintain a running proxy of the distribution using two primary mechanisms:

1. Exponential Moving Averages (EMA)

The most computationally efficient way to track an evolving distribution is to update a running vector of class proportions after every minibatch. For instance, Park et al. and Aimar et al. track the class distribution of unlabeled datasets by maintaining an EMA of the model's soft predictions across minibatches.

In fully test-time adaptation scenarios where data arrives continuously, the authors of Logit-Adjusted Test-Time Adaptation under Partial Class Imbalance use an EMA of batch-wise predictions to dynamically adjust to test streams that only contain a subset of the original training classes. This allows the model to remain highly responsive to sudden environmental changes without storing historical data.

2. Prediction Queues and Memory Banks

An alternative approach relies on maintaining a explicit window of recent history. By utilizing a First-In-First-Out (FIFO) queue of recent model assignments, frameworks like the one proposed by Yang et al. compute a moving average of class frequencies over a fixed horizon.

As the data stream evolves, the queue naturally flushes out stale distribution data. This allows the model to adapt to rapid concept drift or emerging classes without manual weight recalibration, a property also leveraged by Malialis et al. through fixed-length queues dedicated to online resampling.


Why It Works: The Self-Correcting Feedback Loop

Tracking the running distribution is only half the battle; the model must use this estimate to correct its behavior. This is achieved by feeding the estimated prior back into the model's decision boundary, creating a self-correcting feedback loop.

[Incoming Unlabeled Data] ──> [Model Predictions] ──> [Update EMA / Queue]
                                      ▲                         │
                                      │                         ▼
                         [Logit / Weight Adjustment] <── [Estimated Prior]

Logit Adjustment

When a model encounters an imbalanced stream, its raw outputs (logits) are inherently biased toward the training distribution. If we dynamically estimate the target prior as $\hat{p}(y)$, we can apply a logit adjustment. By subtracting $\log \hat{p}(y)$ or adding an adjustment factor directly to the loss function, we penalize over-represented classes and encourage the model to expand its decision boundaries for under-represented ones.

As shown by Xin et al., updating this logit adjustment scheme using an EMA of refurbished labels provides a stable supervision signal even when the incoming data is highly noisy or incorrectly labeled.

Optimal Transport Alignment

Another way to enforce the estimated prior is through constrained optimization. Wang et al. and Shin et al. frame label disambiguation as an Optimal Transport problem. By using algorithms like Sinkhorn-Knopp at inference time, they force the model's cumulative pseudo-labels to match the dynamically estimated prior. This mathematical constraint prevents the model from collapsing into majority-class predictions and ensures that rare "tail" classes retain a distinct margin in the latent feature space (Zhang et al.).


Application Across Learning Paradigms

Dynamic class distribution estimation has become a vital component across several distinct research directions, proving that tracking online priors is universally beneficial when labels are scarce or shifting.

  • Test-Time Adaptation (TTA): When deploying black-box models, Zhou et al. demonstrated theoretically that classification performance under open-world data shift is fundamentally constrained by the accuracy of label distribution estimation. By coupling online trackers with clustering mechanisms, models can adapt to environments where classes appear and disappear over time (Wang et al.; Zhou et al.).
  • Imbalanced Semi-Supervised Learning: In semi-supervised learning, models generate pseudo-labels for unlabeled data based on a small set of labeled examples. If the unlabeled pool is heavily skewed, the model generates biased pseudo-labels, compounding its own errors. Li et al. mitigate this by using momentum-based distribution estimation to track the effective class distribution of participating training samples, correcting the learning objective in real-time as the pseudo-label distribution shifts.
  • Novel Class Discovery (NCD): When a model is tasked with clustering entirely unseen categories, assuming a uniform distribution across clusters fails in real-world, long-tailed settings. Zhang et al. introduce an adaptive imbalance factor that is learned dynamically during the discovery process. This allows the system to estimate the distribution of novel data without any prior knowledge of class counts, stabilizing category identification.

Key Foundational Work

PRIS Kidult Anti-SPAM Solution at the TREC 2005 Spam Track: Improving the Performance of Naive Bayes for Spam Detection (2005) — contributed 2 concept elements to this direction.
Word Sense Disambiguation with Distribution Estimation (2005) — contributed 1 concept elements to this direction.
Active Learning using Dirichlet Processes for Rare Class Discovery and Classification (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamic_class_distribution_estimation_911560c3,
  title = {dynamic class distribution estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-class-distribution-estimation/rdr_911560c3649d44d380655bef816c5ba5},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamic_class_distribution_estimation_911560c3_v1,
  title = {dynamic class distribution estimation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamic-class-distribution-estimation/rdr_911560c3649d44d380655bef816c5ba5/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake