Better Matching, Less Forgetting: A Quality-Guided Matcher for Transformer-based Incremental Object Detection

Association for Artificial Intelligence, De Cheng, Lingyan Ran, Dahu Shi, Peng Wang, Qirui Wu, Yinghui Xing, Shizhou ZhangUnderline 2026 · DOI · OpenAlex

Abstract: Incremental Object Detection (IOD) aims to continuously learn new object classes without forgetting previously learned ones. A persistent challenge is catastrophic forgetting, primarily attributed to background shift in conventional detectors. While pseudo-labeling mitigates this in dense detectors, we identify a novel, distinct source of forgetting specific to DETR-like architectures: background foregrounding. This arises from the exhaustiveness constraint of the Hungarian matcher, which forcibly assigns every ground truth target to one prediction, even when predictions primarily cover background regions (i.e., low IoU). This erroneous supervision compels the model to misclassify background features as specific foreground classes, disrupting learned representations and accelerating forgetting. To address this, we propose a Quality-guided Min-Cost Max-Flow (Q-MCMF) matcher. To avoid forced assignments, Q-MCMF builds a flow graph and prunes implausible matches based on geometric quality. It then optimizes for the final matching that minimizes cost and maximizes valid assignments. This strategy eliminates harmful supervision from background foregrounding while maximizing foreground learning signals. Extensive experiments on the COCO dataset under various incremental settings demonstrate that our method consistently outperforms existing state-of-the-art approaches.

Introduction: The Background Foregrounding Problem

Incremental Object Detection (IOD) is critical for systems that must learn new tasks over time. However, these systems often suffer from catastrophic forgetting. In Transformer-based architectures like DETR, a specific phenomenon called background foregrounding accelerates this loss of knowledge.

Standard DETR models utilize a Hungarian matcher that is exhaustive by design; it must assign every ground-truth object to a prediction. When the model is in the early stages of learning new classes, it often lacks high-quality predictions for those targets. The matcher then forces an assignment to a prediction that may primarily overlap with the background.

Example of background foregrounding where a target is assigned to a low-quality prediction.

This "forced marriage" provides erroneous supervision, effectively teaching the model to recognize background patches as foreground objects. As shown in the statistics below, a significant portion of matches in standard training fall below acceptable Intersection over Union (IoU) thresholds, particularly for new classes.

Percentage of assignments falling below various IoU thresholds for base and new classes.

Proposed Method: Q-MCMF Matcher

To resolve the exhaustiveness trap, we propose the Quality-guided Min-Cost Max-Flow (Q-MCMF) matcher. By reframing bipartite matching as a min-cost max-flow problem on a graph, we introduce the flexibility to leave targets unmatched if they do not meet a quality criterion.

The Q-MCMF process consists of three main steps:

  1. Graph Construction: Predictions and targets are nodes in a flow network.
  2. Quality-guided Edge Pruning: Edges representing potential matches are pruned if their geometric quality (IoU) is below specific thresholds ($\alpha$ for old classes and $\beta$ for new classes).
  3. Dual Optimization: The algorithm solves for a matching that minimizes cost while maximizing the total flow (valid assignments).

This approach prevents the model from receiving toxic supervision from background-heavy predictions, thereby stabilizing the learning process.

Experimental Evaluation

We evaluated Q-MCMF on the COCO 2017 dataset across various incremental protocols, including 70-10 and 40-40 settings, as well as multi-phase and overlapping scenarios.

Key Results

  • Performance Gains: In the 40-40 setting, Q-MCMF improved Average Precision (AP) by 5.1% over the baseline and outperformed the state-of-the-art DCA method by 1.5%.
  • Generalizability: The matcher provided consistent improvements across different DETR variants, including Deformable DETR, DN-DETR, and DAB-DETR.
  • Semantic Preservation: t-SNE visualizations indicated that Q-MCMF preserves distinct semantic clusters for object categories more effectively than standard matchers, which often suffer from feature collapse.

Conclusion

The Q-MCMF matcher addresses a fundamental flaw in how Transformer-based detectors handle incremental learning. By replacing rigid, exhaustive matching with a quality-guided flow optimization, we significantly reduce catastrophic forgetting. Future research will focus on automating the selection of quality thresholds to further enhance model adaptability.

Summary updated

Abstract

Incremental Object Detection (IOD) aims to continuously learn new object classes without forgetting previously learned ones. A persistent challenge is catastrophic forgetting, primarily attributed to background shift in conventional detectors. While pseudo-labeling mitigates this in dense detectors, we identify a novel, distinct source of forgetting specific to DETR-like architectures: \textit{background foregrounding}. This arises from the exhaustiveness constraint of the Hungarian matcher, which forcibly assigns every ground truth target to one prediction, even when predictions primarily cover background regions (i.e., low IoU). This erroneous supervision compels the model to misclassify background features as specific foreground classes, disrupting learned representations and accelerating forgetting. To address this, we propose a Quality-guided Min-Cost Max-Flow (Q-MCMF) matcher. To avoid forced assignments, Q-MCMF builds a flow graph and prunes implausible matches based on geometric quality. It then optimizes for the final matching that minimizes cost and maximizes valid assignments. This strategy eliminates harmful supervision from background foregrounding while maximizing foreground learning signals. Extensive experiments on the COCO dataset under various incremental settings demonstrate that our method consistently outperforms existing state-of-the-art approaches.
BibTeX
@inproceedings{intelligence2026better,
  author = {Association for Artificial Intelligence and De Cheng and Lingyan Ran and Dahu Shi and Peng Wang and Qirui Wu and Yinghui Xing and Shizhou Zhang},
  title = {{Better Matching, Less Forgetting: A Quality-Guided Matcher for Transformer-based Incremental Object Detection}},
  booktitle = {Underline},
  year = {2026},
  doi = {10.48448/xq59-p632},
}
Figure