When machine learning models deploy into the real world, they rarely encounter neat, independent, and identically distributed (IID) data. Instead, they face streaming environments characterized by severe class imbalances, long-tailed distributions, or sequential task shifts.
To prevent models from forgetting past knowledge under these conditions, practitioners often rely on an episodic memory or replay buffer. However, standard buffering strategies—such as uniform reservoir sampling or global First-In, First-Out (FIFO) queues—fail when data is highly skewed. Because their retention mechanics are tied to the stream's natural arrival frequency, majority or recent classes rapidly overwrite minority or historical ones.
Class-balanced memory management solves this by structurally decoupling memory retention from the stream's underlying distribution. By partitioning the buffer along class boundaries, these techniques ensure that rare or historical concepts remain permanently visible to the model.
The Core Mechanism: Class-Partitioned Buffering
The fundamental intuition behind class-balanced memory management is structural isolation. Instead of maintaining a single global queue where samples compete for space, the memory architecture allocates dedicated, independent sub-buffers or slots for each category.
Global FIFO Buffer (Naive):
[ Majority | Majority | Majority | Majority | Minority ] -> (Minority easily pushed out)
Class-Partitioned Buffer (Balanced):
Class A (Majority): [ Sample 1 | Sample 2 | Sample 3 ] -> (High turnover)
Class B (Minority): [ Sample 1 | Sample 2 | Sample 3 ] -> (Long retention)
This structural split fundamentally alters how long data points survive. In Queue-Based Resampling for Online Class Imbalance Learning, Malialis et al. demonstrate that separate per-class FIFO queues naturally perform temporal oversampling. When a common class sample arrives, it only displaces an older sample within its own queue. Conversely, when an exceptionally rare sample arrives, it enters a sparsely updated minority queue, allowing it to persist in memory significantly longer than it would in a global pool.
This preservation is especially critical when memory budgets are tight. Chaudhry et al. showed in Continual Learning with Tiny Episodic Memories that when episodic memory is restricted to just a few samples per category, strict class balancing via per-class FIFO management drastically outperforms random statistical sampling. Similarly, Hu et al. leveraged separate fixed-size buffers in Kernelized Online Imbalanced Learning with Fixed Budgets to ensure that an online learner maintains a representative set of support vectors for both labels, preventing the majority class from overwhelming the decision boundary.
Why It Works: Balancing Gradients and Latent Spaces
Isolating samples into class-specific slots works because it alters the optimization landscape. When a model updates its weights using a standard imbalanced stream, the loss gradients are heavily dominated by the majority class, causing the decision boundary to drift and compress the minority class space.
By drawing training or replay batches exclusively from a class-balanced memory, the model receives a uniform training signal across the entire label space. As Luo et al. observe in Dual-Enhanced Coreset Selection with Class-Wise Collaboration for Online Blurry Class Incremental Learning, dynamically adjusting the buffer to guarantee equal representation for all classes prevents the model from developing severe recency or frequency biases, even when class boundaries are blurred or overlapping.
In representation learning and semi-supervised frameworks, this balance is vital for maintaining the geometric integrity of the latent space:
- Anchor Stability: In SemiGPC: Distribution-Aware Label Refinement for Imbalanced Semi-Supervised Learning Using Gaussian Processes, Lemkhenter et al. partition the memory buffer into distinct sub-buffers per category. This ensures that the label refinement process always has access to a representative set of samples from every class, preventing minority classes from being crowded out of the feature space.
- Preventing Cluster Collapse: When performing contrastive learning on long-tailed data, rare categories risk collapsing into nearby majority clusters because negative pairs from rare classes are seldom sampled. A class-balanced memory ensures that every category can consistently serve as a stable anchor or contrastive target during mini-batch optimization.
Advanced Variations: Dynamic Quotas and Confidence Filters
While strict, equal partitioning provides a robust baseline, real-world data streams often require more flexible memory management strategies to handle concept drift or noisy labels.
Dynamic Allocation
Instead of splitting memory into perfectly equal parts, some architectures dynamically adjust reservoir sizes based on the stream's behavior. As explored by Cano et al. in ROSE: robust online self-adjusting ensemble for continual learning on imbalanced drifting data streams, decoupling memory into independent per-class sliding windows allows for automatic undersampling of majority classes while preserving rare categories that may temporarily disappear and reappear over time. In another approach, Suh et al. show in Long-Tailed Recognition by Mutual Information Maximization between Latent Features and Ground-Truth Labels that maintaining separate memory queues with lengths proportional to sample frequencies can optimize mutual information under long-tailed constraints.
Quality and Confidence Filtering
In scenarios like Test-Time Adaptation (TTA), true labels are missing, forcing the system to rely on pseudo-labels generated by the model itself. This introduces a dangerous failure mode: the model might misclassify a noisy sample, assign it to a minority class buffer, and then continuously train on its own mistake, leading to catastrophic model collapse.
To counter this, the authors of Robust Test-Time Adaptation in Dynamic Scenarios, Yuan et al., introduce a selection heuristic that balances category diversity with prediction certainty and sample recency. By filtering out high-entropy (uncertain) samples before they enter the class-balanced slots, the system ensures that the updates are driven by a reliable and diverse set of examples, neutralizing the tendency of models to overfit to dominant classes or noisy pseudo-labels.
Key Foundational Work
Kernelized Online Imbalanced Learning with Fixed Budgets (2015) — contributed 2 concept elements to this direction.