GCD: Advancing Vision-Language Models for Incremental Object Detection via Global Alignment and Correspondence Distillation

Xu Wang, Zilei Wang, Zihan LinAAAI 2025 · OpenReview

TL;DR

This paper tackles the challenge of incremental object detection (IOD) in vision-language detectors (VLDs), where models must learn new objects without forgetting old ones. They propose GCD, a novel method that addresses catastrophic forgetting by building a unified global semantic structure and using a correspondence mechanism for effective knowledge distillation, achieving state-of-the-art results.

What Problem Does This Paper Address?

Imagine a smart camera that needs to identify objects. Initially, it's trained to recognize cats and dogs. Then, new data arrives, and it needs to learn to identify cars and trucks. The problem is that simply training on the new data often makes the camera forget how to identify cats and dogs – this is called "catastrophic forgetting." This is especially tricky for vision-language detectors (VLDs), which combine visual information with text descriptions. Existing methods often try to learn new tasks in isolation, which breaks down the underlying relationships between objects and their descriptions, leading to forgetting. This paper argues that this "semantic structure collapse" is the root cause of forgetting in VLDs and proposes a new way to maintain both a stable understanding of old knowledge and the ability to learn new knowledge.

Methods

The core of the proposed method, GCD (Global Alignment and Correspondence Distillation), has two main components:

  • Global Alignment: Instead of learning tasks in isolation, GCD integrates knowledge from all previous and current tasks into a single, unified embedding space. This is achieved by aligning new objects with ground truth labels and old objects with pseudo-labels. This global perspective helps build a cohesive semantic structure.
  • Correspondence Distillation: To effectively transfer knowledge from a "teacher" model (trained on previous tasks) to a "student" model (learning new tasks), GCD introduces a Semantic Correspondence Mechanism (SCM). This mechanism ensures that the teacher and student generate consistent proposals and decodings, even when dealing with different text prompts.
  • Correspondence Response Distillation (CRD): This part distills the teacher's predictions (logits and bounding boxes) to the student, paying special attention to preserving the performance of less common (weak) categories.
  • Correspondence Topology Distillation (CTD): This goes a step further by ensuring that the relationships between object prototypes and text prototypes in the student model remain consistent with those in the teacher model, thus preserving the local semantic structure.

Datasets

The experiments were conducted on the COCO 2017 dataset, a standard benchmark for object detection tasks. The paper uses various two-phase and multi-phase settings for incremental learning, splitting the dataset into different sets of classes for sequential training.

Key Insights and Ideas

  • Semantic Structure Collapse is the Culprit: The paper's central insight is that catastrophic forgetting in VLDs isn't just about forgetting specific objects, but about the breakdown of the underlying relationships between visual features and textual descriptions.
  • Global vs. Local Alignment: The authors highlight the limitations of purely "local alignment" (learning tasks in isolation) and advocate for a "global alignment" approach that integrates knowledge across all tasks.
  • The Need for a "Correspondence" Mechanism: Simply distilling knowledge isn't enough for VLDs. A mechanism that ensures consistency in how objects and text are processed (like SCM) is crucial for effective knowledge transfer.
  • Balancing Stability and Plasticity: GCD aims to strike a balance between maintaining stable knowledge of old classes (stability) and effectively learning new classes (plasticity), a fundamental challenge in continual learning.

Key Results

GCD achieves state-of-the-art performance on the COCO 2017 dataset across various incremental object detection scenarios. For instance, in the 70+10 setting (70 old classes, 10 new classes), GCD achieved an AP of 46.7%, outperforming previous methods. In multi-phase settings, GCD also showed significant improvements, demonstrating its ability to handle more complex incremental learning scenarios. The ablation studies confirm the effectiveness of each component of GCD, with CTD playing a key role in reducing forgetting (FPP of 1.9%).

Limitations and Future Work

The paper acknowledges that while GCD significantly improves performance, the generation of pseudo-labels can still suffer from information redundancy and noise, potentially leading to overconfident predictions that can disrupt optimization. Future work could explore more robust pseudo-label generation techniques or alternative methods to mitigate these issues. The authors also suggest that their framework could be extended to other vision-language tasks beyond object detection.

Summary updated

Abstract

Incremental object detection (IOD) is a challenging task that requires detection models to continuously learn from newly arriving data. This work focuses on incremental learning for vision-language detectors (VLDs), an under explored domain. Existing research typically adopts a local alignment paradigm to avoid label conflicts, where different tasks are learned separately without interaction. However, we reveal that this practice fails to effectively preserve the semantic structure. Specifically, aligned relationships between objects and texts would collapse when handling novel categories, ultimately leading to catastrophic forgetting. Though knowledge distillation (KD) is a common approach for tackling this, traditional KD performs poorly when directly applied to VLDs, as for different phases, a natural knowledge gap exists in both encoding and decoding processes. To address above issues, we propose a novel method called Global alignment and Correspondence Distillation (GCD). Differently, we first integrate knowledge across phases within the same embedding space to construct global semantic structure. We then enable effective knowledge distillation in VLDs through a semantic correspondence mechanism, ensuring consistent proposal generation and decoding. On the top of that, we distill teacher model’s informative predictions and topological relationships to maintain stable local semantic structure. Extensive experiments on COCO 2017 demonstrate that our method significantly outperforms existing approaches, achieving new state-of-the-art in various IOD scenarios.
BibTeX
@inproceedings{wang2025gcd,
  author = {Xu Wang and Zilei Wang and Zihan Lin},
  title = {{GCD: Advancing Vision-Language Models for Incremental Object Detection via Global Alignment and Correspondence Distillation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=4CgHrf2NPn},
}
Figure