DIODE: Dilatable Incremental Object Detection

Can Peng, Kun Zhao, Sam Maksoud, Tianren Wang, Brian C. LovellarXiv 2021 · OpenReview

TL;DR

This paper introduces DIODE, a novel approach to incremental object detection that tackles the "catastrophic forgetting" problem. DIODE combines a constrained regularization technique (Constrained EWC) to preserve old knowledge with a "dilatable" network architecture that expands slightly to learn new tasks, achieving significant performance improvements over existing methods.

What Problem Does This Paper Address?

Deep learning models excel at learning specific tasks but struggle to adapt to new information without forgetting what they've already learned. This is particularly problematic in real-world scenarios where new objects or categories emerge over time, requiring continuous learning. Conventional methods for incremental object detection often suffer from a performance decay in later learning stages, creating a trade-off between preserving old knowledge and learning new tasks. This paper addresses the challenge of multi-step incremental object detection, where a model must learn a sequence of new object categories while retaining its ability to detect previously learned ones, even when new data lacks annotations for older categories.

Methods

DIODE proposes a hybrid approach to incremental object detection, building upon the FCOS (Fully Convolutional One-Stage Object Detection) framework. The core innovations are:

  • Constrained Elastic Weight Consolidation (EWC): Instead of applying EWC to all parameters, DIODE adaptively penalizes changes to important weights in the backbone network (ResNet, FPN, and the classification head's convolutional tower). This is based on the observation that certain parameters (especially in detection heads) have disproportionately large importance values, which can lead to gradient explosion and hinder effective regularization. By focusing EWC on the backbone, DIODE aims to preserve crucial features for old tasks more effectively.
  • Dilatable Module: To overcome the rigidity introduced by EWC and promote new task learning, DIODE employs a "dilatable" module. This involves adding a limited number of task-specific parameters (specifically, convolutional layers with a kernel size of one) to adapt feature maps from the FPN and the classification head's convolutional tower. This expansion is introduced from the second incremental learning step onwards, allowing the model to leverage existing parameters before introducing new ones.
  • Pseudo-Annotation Strategy: To mitigate the issue of missing annotations for old classes in new task data, DIODE utilizes pseudo-annotations. These are generated by the model from previous learning stages, effectively providing a form of "self-supervision" for old classes on new data.

Datasets

The proposed method was evaluated on two widely used object detection benchmarks:

  • PASCAL VOC 2007: Used for training and validation, with the test set for evaluation.
  • COCO 2014: Used for training and validation (minus minival), with the minival set for evaluation.

The paper explores various incremental learning protocols, including adding multiple classes at once (e.g., five classes) and adding classes one by one.

Key Insights and Ideas

  • Focusing EWC is Crucial: Applying regularization techniques like EWC indiscriminately can be detrimental. DIODE's insight is that by strategically applying EWC to the backbone and feature extractor, rather than all detection-specific heads, old knowledge preservation becomes more effective.
  • Dynamic Architecture for Plasticity: A fixed network architecture struggles with continuous learning. DIODE's "dilatable" module, which adds a small number of task-specific parameters, provides the necessary plasticity to learn new tasks without completely overwriting old knowledge.
  • Pseudo-Annotations are a Strong Baseline: The paper highlights that even a simple pseudo-annotation strategy significantly boosts performance for regularization-based methods, underscoring the importance of addressing missing annotations.
  • Incremental Learning Difficulty Varies: The authors note that the performance gains of their method can vary depending on the incremental setting, suggesting that factors like data amount, annotation scarcity, class similarity, and the ratio of old to new classes play a significant role.

Key Results

DIODE demonstrates substantial improvements over state-of-the-art incremental object detection methods across various multi-step learning scenarios on both VOC and COCO datasets.

  • VOC Dataset:
    • In a three-step incremental scenario (adding five classes at a time), DIODE achieved an average improvement of 2.3% mAP over IncDet.
  • In a four-step scenario (adding four classes at a time), DIODE outperformed IncDet by 4.8% and 5.0% mAP in later steps, showing robustness.
  • Across various settings, DIODE consistently outperformed online EWC and IncDet, with improvements of up to 44.4% and 3.0% mAP respectively in some cases.
  • COCO Dataset:
    • In a ten-class-at-a-time protocol, DIODE improved IncDet's performance by an average of 3.1% mAP (at 0.5 IoU).
  • In a five-class-at-a-time protocol, DIODE showed consistent improvements over IncDet.
  • Parameter Efficiency: Notably, DIODE achieves these improvements by increasing the number of parameters by a modest amount, for example, only 1.2% for each newly learned task in one scenario.

Limitations and Future Work

  • GPU Limitations: The authors mention that due to GPU limitations, they were unable to conduct experiments with very large-scale multi-step scenarios.
  • Hyperparameter Sensitivity: Like many EWC-based methods, DIODE's performance is sensitive to the regularization coefficient (λ). The paper discusses finding optimal λ values by identifying the point just before gradient explosion.
  • Future Work: The paper implicitly suggests that further exploration of optimal ways to balance regularization and architectural expansion for different incremental learning complexities could be beneficial. The authors also note that while their method significantly improves performance, further research into more robust and efficient methods for handling the trade-off between plasticity and stability in incremental learning is warranted.

Summary updated

Abstract

To accommodate rapid changes in the real world, the cognition system of humans is capable of continually learning concepts. On the contrary, conventional deep learning models lack this capability of preserving previously learned knowledge. When a neural network is fine-tuned to learn new tasks, its performance on previously trained tasks will significantly deteriorate. Many recent works on incremental object detection tackle this problem by introducing advanced regularization. Although these methods have shown promising results, the benefits are often short-lived after the first incremental step. Under multi-step incremental learning, the trade-off between old knowledge preserving and new task learning becomes progressively more severe. Thus, the performance of regularization-based incremental object detectors gradually decays for subsequent learning steps. In this paper, we aim to alleviate this performance decay on multi-step incremental detection tasks by proposing a dilatable incremental object detector (DIODE). For the task-shared parameters, our method adaptively penalizes the changes of important weights for previous tasks. At the same time, the structure of the model is dilated or expanded by a limited number of task-specific parameters to promote new task learning. Extensive experiments on PASCAL VOC and COCO datasets demonstrate substantial improvements over the state-of-the-art methods. Notably, compared with the state-of-the-art methods, our method achieves up to 6.0% performance improvement by increasing the number of parameters by just 1.2% for each newly learned task.
BibTeX
@misc{peng2021diode,
  author = {Can Peng and Kun Zhao and Sam Maksoud and Tianren Wang and Brian C. Lovell},
  title = {{DIODE: Dilatable Incremental Object Detection}},
  year = {2021},
  eprint = {2108.05627},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.05627},
}
Figure