In standard object detection, any region of an image not explicitly labeled with a bounding box is considered "background." While this assumption works for static datasets, it becomes a primary driver of catastrophic forgetting in incremental learning. When a model is trained to recognize new classes (e.g., "dogs") using images that happen to contain previously learned classes (e.g., "cats") that are no longer labeled, the training process treats those cats as negative background samples. This "background shift" actively teaches the model to suppress its existing knowledge.
The Missing Annotation Problem
The core difficulty in incremental object detection, as identified by Cordelia Schmid, Konstantin Shmelkov, and Karteek Alahari in Incremental Learning of Object Detectors without Catastrophic Forgetting, is that new training sets are rarely exhaustive.
In a typical incremental scenario, a model first learns Task A and then moves to Task B. The images in Task B are labeled only for the new categories. However, because objects co-occur in the real world, Task B images frequently contain Task A objects. If the training loop treats these unlabeled Task A objects as part of the background, the gradient descent process penalizes the model for detecting them. This creates a "negative signal" that forces the network to unlearn the features it previously mastered.
Modeling the Background as Unknown
To combat this, researchers have moved away from treating the background as a monolithic "empty" class. Fabio Cermelli and colleagues introduced the MMA (Modeling Missing Annotations) framework in Modeling Missing Annotations for Incremental Learning in Object Detection.
The intuition behind MMA is to redesign the loss function to acknowledge uncertainty. Instead of a hard classification between "new class" and "background," the model treats unlabeled regions as potentially containing either background or previously learned objects. By relaxing the penalty on detections in these ambiguous regions, the model avoids the aggressive suppression of old knowledge. Similarly, Shiyu Chang and David D. Cox proposed Pseudo-Positive-Aware Sampling (PPAS) in Lifelong Object Detection, which identifies potential instances of old classes in new data and prevents them from being used as negative background samples during training.
Label Propagation and Self-Distillation
Another dominant strategy involves using the model's own history to fill in the gaps. Since the model already knows how to detect Task A objects, it can be used to "pseudo-label" the Task B images before training begins.
Riccardo De Monte and Gian Antonio Susto demonstrate this in Replay Consolidation with Label Propagation for Continual Object Detection. By propagating labels from the old model version to the new training set, the "missing" annotations are restored, turning the background shift back into a standard supervised learning problem.
For modern, one-stage detectors like YOLO, Riccardo De Monte also proposed YOLO LwF in Teach YOLO to Remember: A Self-Distillation Approach for Continual Object Detection. This approach uses distillation losses—where the current model tries to match the prediction confidence and bounding box coordinates of its predecessor—to ensure that the "background" in new images doesn't overwrite the high-confidence detections of the past.
Generative Replay and Hallucinated Reminders
When storing old images is not possible due to privacy or memory constraints, researchers use generative models to "remind" the detector of what old classes look like.
In SDDGR: Stable Diffusion-Based Deep Generative Replay for Class Incremental Object Detection, Junsu Kim and Hoseong Cho leverage Stable Diffusion to generate synthetic training data for old classes. This generative replay ensures the model always sees a balanced diet of old and new objects, preventing the background of new tasks from dominating the feature space. A similar approach, DiffusePast, was applied to semantic segmentation by Jingfan Chen and Zhaoxiang Zhang in DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation, using diffusion models to produce structure-preserved synthetic images that provide high-quality pseudo-labels for incremental tasks.
Architectural Solutions
Beyond loss functions and data augmentation, some researchers suggest that the detector's architecture itself can mitigate forgetting. Arnaud Dapogny and Kevin Bailly introduced MultIOD in MultIOD: Rehearsal-free Multihead Incremental Object Detector. By using a multi-head architecture where each head is responsible for a specific set of tasks, the model can freeze the parameters associated with old classes. This physical separation prevents the gradients from new tasks—and the associated background shift—from corrupting the weights that define previously learned categories.