In standard class-incremental learning (CIL), a model learns to recognize new categories over time from a sequential stream of data. For simple image classification tasks, this process is straightforward: an image contains a single object, and the model is given a single label. However, when we move to dense prediction tasks—such as semantic segmentation, object detection, and named entity recognition (NER)—the real world introduces a severe structural complication.
In a dense scene, multiple objects co-occur. When a dataset is curated to teach a model a new set of classes (Task $t$), instances of previously learned classes (Task $<t$) or future classes (Task $>t$) are often physically present in the training images or text. Because they are not the focus of the current task, they lack explicit annotations and are lumped into a single, catch-all category: the background (or the "non-entity" class in NLP).
This phenomenon is known as background shift or semantic label ambiguity. It introduces a destructive, conflicting training signal that fundamentally breaks standard continual learning algorithms.
The Mechanism of Forgetting: Conflicting Supervision
To understand why background shift causes catastrophic forgetting, consider what happens during an incremental update. To prevent a model from forgetting past knowledge, standard continual learning frameworks employ knowledge distillation (KD). The distillation loss penalizes the new model if its predictions deviate from the old model's beliefs.
When an image in a new training phase contains an unannotated object from a previous category, the model faces an algorithmic contradiction:
- The Current Task Loss (e.g., Cross-Entropy): Sees that the object is unlabeled, treats it as part of the negative background class, and penalizes the model for predicting anything else.
- The Distillation Loss: Remembers the old model's training, recognizes the object, and penalizes the model if it fails to predict the old class.
As Bernt Schiele et al. demonstrate in Continual Detection Transformer for Incremental Object Detection, this creates a direct optimization conflict. The network is actively taught to suppress its existing knowledge, treating valid historical categories as negative noise. Pietro Zanuttigh argues in Knowledge distillation for incremental learning in semantic segmentation that this spatial co-occurrence makes dense prediction uniquely challenging; updating parameters for new categories inherently degrades the model's ability to segment known objects in the same scene.
Standard CIL: [Image of Dog] -> Label: Dog (Clean separation)
Dense CIL: [Image of Cat and Dog]
Current Task Label: Cat
Unlabeled Dog -> Re-mapped to "Background"
Result: Model is penalized for recognizing the Dog.
This problem is not confined to computer vision. In Class-Incremental NER, tokens belonging to past or future entity types are routinely mislabeled as the generic "O" (Non-entity) class. Ruotian Ma et al. show in Learning "O" Helps for Learning More: Handling the Unlabeled Entity Problem for Class-incremental NER that performance degradation in sequential sequence labeling is primarily driven by this specific confusion with the background class, rather than a loss of the features that define the classes themselves.
Redefining the Background
The core insight behind modern solutions is that the background cannot be treated as a monolithic negative class. Instead, models must explicitly account for the hidden semantic layers within the unannotated space.
Modeling the Unknown
One prominent strategy is to isolate potential target classes from the true background. Sungmin Cha et al. pioneered this in SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning by introducing an explicit "Unknown" class. By shifting the classification layer from a standard Softmax (which forces classes to compete) to a Sigmoid-based binary cross-entropy loss, they allowed the model to remain agnostic about unannotated regions rather than forcing them to be negative samples.
Proactive Mining
Rather than waiting for future classes to cause conflicts, some approaches proactively mine them from the background. Zekang Zhang proposes this in Mining Unseen Classes via Regional Objectness: A Simple Baseline for Incremental Segmentation. By using a class-agnostic proposal generator, the framework identifies regions with high "objectness" within the background and clusters them into temporary "unseen" categories. This prevents the model from developing a destructive bias against those features before they are formally introduced.
Calibrating the Optimization Signal
Another major front in mitigating background shift involves modifying the loss functions and distillation targets to tolerate shifting label definitions.
Label-Guided Alignment
Ze Yang et al. demonstrate in Label-Guided Knowledge Distillation for Continual Semantic Segmentation on 2D Images and 3D Point Clouds that resolving the background-to-novel class conflict is the single largest driver of performance gains in incremental segmentation. Their framework uses ground-truth labels to dynamically mask and align the probability distributions between the old and new models, ensuring that distillation only occurs where labels are semantically consistent.
Parameter-Efficient Adjustments
When updating models via parameter-efficient tuning, such as low-rank adaptation, the background shift can quickly corrupt the shared parameter space. The authors of CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation address this by modifying the underlying loss formulations to explicitly account for the shifting definition of the background, preventing the low-rank adapters from overfitting to the noisy negative signals of the current task.
Contextual Decoupling
Because objects are deeply tied to their surroundings, background shift also introduces contextual bias. Hanbin Zhao et al. address this in RBC: Rectifying the Biased Context in Continual Semantic Segmentation by utilizing "image duplets" and a contextual consistency loss. This explicitly decouples the object representations from their shifting backgrounds, ensuring that the model's internal features remain stable even when the surrounding semantic labels change.
Ubiquity Across Modalities and Frameworks
The challenge of background shift is a universal property of dense incremental learning, manifesting across diverse domains:
- Multi-Label Learning: In multi-label incremental settings, images naturally contain multiple overlapping categories. Thomas De Min in Less is more: Summarizing Patch Tokens for efficient Multi-Label Class-Incremental Learning and Xiang Zhang in L3A: Label-Augmented Analytic Adaptation for Multi-Label Class Incremental Learning show that unlabelled co-occurring objects from previous tasks trigger an unintended suppression of prior knowledge, requiring task-specific pathways or label-augmented adaptations to bypass the background noise.
- Federated Learning: The problem even extends to distributed environments. Jiaxu Miao demonstrates in FedSeg: Class-Heterogeneous Federated Learning for Semantic Segmentation that when individual clients only possess labels for a subset of classes, their local background categories act as a source of noise that actively suppresses valid features present in other clients' datasets, causing severe model drift during global aggregation.
Ultimately, resolving background shift requires moving away from the assumption that an absence of a label implies an absence of an object. By treating the background as a dynamic, semantically rich mixture of past, present, and future classes, these frameworks allow deep networks to safely learn in complex, multi-object environments without unlearning the past.