Abstract
The continual learning problem has been widely studied in image classification, while rare work has been explored in object detection. Some recent works apply knowledge distillation to constrain the model to retain old knowledge, but this rigid constraint is detrimental for learning new knowledge. In our paper, we propose a new scheme for continual learning of object detection, namely Contrast R-CNN, an approach strikes a balance between retaining the old knowledge and learning the new knowledge. Furthermore, we design a Proposal Contrast to eliminate the ambiguity between old and new instance to make the continual learning more robust. Extensive evaluation on the PASCAL VOC dataset demonstrates the effectiveness of our approach.
Introduction
Deep learning models often suffer from "catastrophic forgetting," where fine-tuning on new classes causes the model to lose its ability to recognize previously learned categories. This challenge is particularly acute in object detection because annotations for "old" classes are typically absent in new training phases. Traditional solutions rely on Knowledge Distillation (KD) to force a new model to mimic a teacher model trained on old data. However, the authors argue that KD imposes overly rigid constraints, creating a performance imbalance where the detector excels at old tasks but fails to learn new features effectively. This "distillation gap" can lead to a performance disparity of up to 15% mAP between old and new classes.
Methodology: Contrast R-CNN
To address these limitations, the authors introduce Contrast R-CNN, a framework designed to balance stability (retaining old knowledge) and plasticity (learning new knowledge). The architecture consists of two main components: Entropy-based Data Distillation and Temporal Contrastive Learning.
Entropy-based Data Distillation
To mitigate the lack of old-class annotations, the previous model generates pseudo-labels on the new dataset. To ensure quality, a Median Entropy Filter is applied, using category-specific thresholds based on prediction confidence. Furthermore, Uncertainty Weighting scales the loss based on the model's confidence in a distilled bounding box, prioritizing high-quality proposals.
Temporal Contrastive Learning
The core innovation is the Temporal Contrast R-CNN. It maintains a "New" model updated via standard backpropagation and a "Temporal" model updated through a momentum-based moving average.
- Proposal Contrast Head: A sub-network maps Region of Interest (RoI) features into an embedding space.
- Contrastive Loss: A supervised contrastive approach encourages the model to cluster embeddings of the same category while pushing different categories apart in the feature space.
- Memory Bank: A queue-based bank stores embeddings from previous mini-batches, providing a diverse set of negative samples to stabilize the learning process.
Experimental Results
The proposed method was evaluated on the PASCAL VOC 2007 dataset across three incremental learning scenarios: 10+10, 15+5, and 19+1 classes.
As shown in the results table, Contrast R-CNN significantly outperforms the previous state-of-the-art (Faster ILOD) in learning new categories. In the 10+10 setting, it achieved 63.44% mAP on new classes, a substantial improvement over Faster ILOD's 53.10%. While distillation-heavy methods might retain slightly more performance on old classes in the 19+1 setting, Contrast R-CNN provides a much more balanced overall performance. Ablation studies confirmed that the Memory Bank is crucial, with its removal leading to a drop of up to 3.76% mAP.
Conclusion and Future Work
Contrast R-CNN demonstrates that learning a discriminative feature space is more effective for continual object detection than rigid output mimicry. By enlarging decision boundaries between classes, the model effectively distinguishes between old and new instances. Future research will focus on scaling this approach to larger datasets like MS COCO and exploring multi-step continual learning scenarios to address long-term "concept drift."
Summary updated