MuSCM: Mutual Spatial Correlation Mapping for Class Incremental Detection Transformer

Jian Zhong, Yifan Jiao, Xi Shao, Bing-Kun BaoCVPR 2026 Findings 2026 · OpenReview

TL;DR

Class Incremental Object Detection (CIOD) in DETR models often suffers from catastrophic forgetting because standard knowledge distillation only aligns final predictions, ignoring where the model is looking. This paper introduces MuSCM (Mutual Spatial Correlation Mapping), a framework that decouples final predictions from spatial attention maps and mutually maps these spatial correlations between old and new models. This approach achieves state-of-the-art performance on COCO 2017 without needing to store or replay old images.

What Problem Does This Paper Address?

When deep learning models are trained sequentially on new classes, they suffer from catastrophic forgetting—they completely forget how to detect objects from previous training stages. Class Incremental Object Detection (CIOD) tries to solve this.

For modern DEtection TRansformers (DETR), the standard remedy is knowledge distillation: making a new "student" model mimic the final predictions (bounding boxes and classifications) of the old "teacher" model.

However, the authors point out a major flaw in this approach: prediction-level alignment is deceptive. Two models can output the exact same bounding box prediction while their internal object queries are looking at completely different local features. Because standard distillation ignores these underlying spatial correlations, the student model fails to truly inherit the teacher's foundational knowledge of where to focus, making the distillation process fragile and unreliable.

Figure 1. (a) We observe that the predictions aligned by existing methods may stem from object queries that aggregate highly dis- parate local features. (b) The core of our method lies in acquiring spatially consistent queries via mutual mapping of spatial correla- tions, thereby facilitating subsequent distillation.

Methods

To bridge this spatial gap, the authors propose MuSCM, which decouples spatial correlations from final predictions and distills them via two main modules:

  1. Spatially Consistent Distillation (SCD): In incremental learning, the student model inherits its exact architecture from the teacher. The authors exploit this structural consistency to perform a lossless mutual mapping. SCD takes the spatial attention maps (sampling offsets and attention weights) from the student and projects them into the teacher, and vice versa. This forces both models to generate "mapped queries" that look at the exact same image regions, ensuring that the subsequent prediction distillation is perfectly aligned spatially. It also distills the relationships between different queries.
  2. Spatial Correlation Matching (SCM): SCM explicitly matches queries between the teacher and student using a novel bipartite matching cost. This cost combines both prediction similarity and the similarity of their spatial sampling points. Once matched, it forces the student queries to adopt the teacher's exact spatial attention patterns.
Figure 2. The overview pipeline of proposed MuSCM, which consists of Spatially Consistent Distillation (SCD) and Spatial Correlation Matching (SCM). We employ MuSCM to each transformer layer in decoder. We mutually map the spatial correlation between models, acquiring two pairs of spatially consistent queries for SCD. We also distill the inter-query relations based on queries output by the previous layer. The SCM distills teacher’s spatial correlation to student through bipartite matching.

Datasets

The framework was evaluated on the standard MS COCO 2017 dataset, which contains 80 diverse object classes. The authors evaluated performance across several standard incremental splits:

  • Single-step increments: 70 base classes + 10 new classes, and 40 base classes + 40 new classes.
  • Multi-step increments: 40 base classes followed by 4 steps of 10 classes (40+10+10+10+10) or 2 steps of 20 classes (40+20+20).

Key Insights and Ideas

  • Exploiting Structural Sameness: Unlike standard knowledge distillation (where a small student learns from a completely different, larger teacher), incremental learning features a student and teacher with identical architectures. MuSCM uniquely leverages this to pass complex attention maps back and forth losslessly.
  • Decoupling is Crucial: Final predictions and spatial attention maps represent different types of knowledge. Trying to optimize both simultaneously without decoupling causes friction. Separating them allows for cleaner, more effective knowledge transfer.
  • Beating Data Replay Without Data: Historically, "exemplar replay" methods (which save a subset of real images from old classes to re-train the model) outperformed "replay-free" methods. MuSCM flips this script by proving that high-fidelity structural distillation can actually outperform raw data rehearsal.

Key Results

MuSCM achieves state-of-the-art results on COCO 2017 benchmarks, consistently outperforming both replay-free and replay-based methods:

  • 70+10 Split: MuSCM achieved 42.5% AP, outperforming the previous best replay-free method (DyQ-DETR) by 3.0% AP, and even beating its replay-based counterpart (42.4% AP).
  • 40+40 Split: It achieved 43.5% AP, surpassing the state-of-the-art replay-based method SDDGR by 0.5% AP.
  • Multi-Step Scenarios: In the grueling 4-step incremental setting (40+10×4), where catastrophic forgetting is usually most severe, MuSCM maintained a final AP of 36.3%, outperforming the best replay-based method by 2.5% AP.

Limitations and Future Work

Despite its strong performance, the authors noted some failure cases in fine-grained classification. For instance, the model occasionally struggled to distinguish highly confusing, context-dependent classes (like confusing a "couch" with a "bed") due to receptive field limitations.

The authors suggest that future work should focus on expanding the model's receptive field or incorporating better global context to help resolve these ambiguous boundaries. Additionally, while validated on Deformable DETR and DN-DETR, exploring how this spatial mapping generalizes to newer, real-time transformer architectures (like RT-DETR) remains an open direction.

Summary updated

Abstract

Class Incremental object detection (CIOD) aims to train object detectors continuously while alleviating catastrophic forgetting of previously learned knowledge. The DEtection TRansformer (DETR) has largely advanced object detection due to its simplicity and efficacy. Existing CIOD methods typically employ knowledge distillation for DETR, aligning the current model's predictions with those of the previous model to preserve knowledge. However, we observe that such prediction-level alignment can be unreliable, since aligned predictions may stem from object queries that aggregate distinct local features, indicating a failure to preserve the spatial correlations between queries and local features. To address this, we propose the Mutual Spatial Correlation Mapping (MuSCM) framework, comprising the Spatially Consistent Distillation (SCD) and Spatial Correlation Matching (SCM). Firstly, SCD decouples spatial correlations from predictions and maps them mutually between models, providing spatially consistent queries for prediction alignment. Secondly, the SCM module establishes correspondences between the queries of the two models based on correlation similarity and enforces the spatial consistency of the matched queries. Extensive experiments on COCO 2017 demonstrate that MuSCM achieves state-of-the-art performance in the CIOD, even outperforming replay-based methods.
BibTeX
@inproceedings{zhong2026muscm,
  author = {Jian Zhong and Yifan Jiao and Xi Shao and Bing-Kun Bao},
  title = {{MuSCM: Mutual Spatial Correlation Mapping for Class Incremental Detection Transformer}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=63OESYMKYL},
}
Figure