TL;DR
Class Incremental Object Detection (CIOD) in DETR models often suffers from catastrophic forgetting because standard knowledge distillation only aligns final predictions, ignoring where the model is looking. This paper introduces MuSCM (Mutual Spatial Correlation Mapping), a framework that decouples final predictions from spatial attention maps and mutually maps these spatial correlations between old and new models. This approach achieves state-of-the-art performance on COCO 2017 without needing to store or replay old images.
What Problem Does This Paper Address?
When deep learning models are trained sequentially on new classes, they suffer from catastrophic forgetting—they completely forget how to detect objects from previous training stages. Class Incremental Object Detection (CIOD) tries to solve this.
For modern DEtection TRansformers (DETR), the standard remedy is knowledge distillation: making a new "student" model mimic the final predictions (bounding boxes and classifications) of the old "teacher" model.
However, the authors point out a major flaw in this approach: prediction-level alignment is deceptive. Two models can output the exact same bounding box prediction while their internal object queries are looking at completely different local features. Because standard distillation ignores these underlying spatial correlations, the student model fails to truly inherit the teacher's foundational knowledge of where to focus, making the distillation process fragile and unreliable.
Methods
To bridge this spatial gap, the authors propose MuSCM, which decouples spatial correlations from final predictions and distills them via two main modules:
- Spatially Consistent Distillation (SCD): In incremental learning, the student model inherits its exact architecture from the teacher. The authors exploit this structural consistency to perform a lossless mutual mapping. SCD takes the spatial attention maps (sampling offsets and attention weights) from the student and projects them into the teacher, and vice versa. This forces both models to generate "mapped queries" that look at the exact same image regions, ensuring that the subsequent prediction distillation is perfectly aligned spatially. It also distills the relationships between different queries.
- Spatial Correlation Matching (SCM): SCM explicitly matches queries between the teacher and student using a novel bipartite matching cost. This cost combines both prediction similarity and the similarity of their spatial sampling points. Once matched, it forces the student queries to adopt the teacher's exact spatial attention patterns.
Datasets
The framework was evaluated on the standard MS COCO 2017 dataset, which contains 80 diverse object classes. The authors evaluated performance across several standard incremental splits:
- Single-step increments: 70 base classes + 10 new classes, and 40 base classes + 40 new classes.
- Multi-step increments: 40 base classes followed by 4 steps of 10 classes (40+10+10+10+10) or 2 steps of 20 classes (40+20+20).
Key Insights and Ideas
- Exploiting Structural Sameness: Unlike standard knowledge distillation (where a small student learns from a completely different, larger teacher), incremental learning features a student and teacher with identical architectures. MuSCM uniquely leverages this to pass complex attention maps back and forth losslessly.
- Decoupling is Crucial: Final predictions and spatial attention maps represent different types of knowledge. Trying to optimize both simultaneously without decoupling causes friction. Separating them allows for cleaner, more effective knowledge transfer.
- Beating Data Replay Without Data: Historically, "exemplar replay" methods (which save a subset of real images from old classes to re-train the model) outperformed "replay-free" methods. MuSCM flips this script by proving that high-fidelity structural distillation can actually outperform raw data rehearsal.
Key Results
MuSCM achieves state-of-the-art results on COCO 2017 benchmarks, consistently outperforming both replay-free and replay-based methods:
- 70+10 Split: MuSCM achieved 42.5% AP, outperforming the previous best replay-free method (DyQ-DETR) by 3.0% AP, and even beating its replay-based counterpart (42.4% AP).
- 40+40 Split: It achieved 43.5% AP, surpassing the state-of-the-art replay-based method SDDGR by 0.5% AP.
- Multi-Step Scenarios: In the grueling 4-step incremental setting (40+10×4), where catastrophic forgetting is usually most severe, MuSCM maintained a final AP of 36.3%, outperforming the best replay-based method by 2.5% AP.
Limitations and Future Work
Despite its strong performance, the authors noted some failure cases in fine-grained classification. For instance, the model occasionally struggled to distinguish highly confusing, context-dependent classes (like confusing a "couch" with a "bed") due to receptive field limitations.
The authors suggest that future work should focus on expanding the model's receptive field or incorporating better global context to help resolve these ambiguous boundaries. Additionally, while validated on Deformable DETR and DN-DETR, exploring how this spatial mapping generalizes to newer, real-time transformer architectures (like RT-DETR) remains an open direction.
Summary updated