nVFNet-RDC: Replay and Non-Local Distillation Collaboration for Continual Object Detection

Jinxiang Lai, Wenlong Liu, Jun Liu

TL;DR

The paper presents nVFNet-RDC, the winning solution for the CVPR 2022 CLVision Challenge (Tracks 2 and 3). By combining a specialized "Non-Local" detector architecture with a collaborative strategy of experience replay and relation-based distillation, the authors achieved state-of-the-art performance in continual category-level and instance-level object detection.

What Problem Does This Paper Address?

The core challenge is Continual Learning (CL) for object detection. In a typical machine learning setup, a model is trained on a static dataset. In CL, the model must learn from a stream of data (tasks) over time. The primary hurdle is catastrophic forgetting: as the model learns new categories or instances, it tends to overwrite the weights used to recognize previous ones.

This paper specifically tackles this in the context of the EgoObjects Dataset, which consists of egocentric (first-person) videos. This adds complexity due to varying lighting, camera motion, and the need to distinguish between broad categories (Track 2) and specific individual objects (Track 3).

Methods

The authors propose a framework called nVFNet-RDC, which stands for nVFNet (the detector) and RDC (Replay and Distillation Collaboration).

  1. nVFNet (Non-Local VarifocalNet): The authors started with VarifocalNet (VFNet) as their base one-stage detector. They modified it by inserting a Non-Local Module into the dense classifier. This module allows the network to model relationships between different pixel-wise instances in an image, improving classification accuracy.
  2. Replay Strategy: To combat forgetting, they maintain a "replay buffer" (3,500 samples for Track 2; 5,000 for Track 3). They use Video-wise Random Sampling, ensuring that the buffer contains a representative set of samples from past videos to remind the model of what it previously learned.
  3. Non-Local Distillation: This is the "secret sauce." In a teacher-student setup, the "teacher" (the model from the previous task) helps the "student" (the current model). Instead of just distilling raw feature maps, they distill the relation features from the Non-Local Module. The authors argue this "weak feature distillation" is more effective because it focuses on the discriminative relationships between objects rather than just raw pixel values, which might change between tasks.
  4. Backbone & Augmentation: They utilized a powerful Res2Net101-DCN backbone and applied PhotoMetricDistortion for data augmentation.

Datasets

The primary dataset used is EgoObjects, provided by Meta.

  • Scale: ~70,000 training images and 10,000 testing images.
  • Nature: Egocentric videos depicting common household/workplace objects.
  • Complexity: Features high variability in lighting and scale, though scene complexity (number of objects) is generally lower than COCO.
  • Tracks:
    • Track 2: Class-incremental category-level detection (277 total categories).
  • Track 3: Class-incremental instance-level detection (1,110 total instances).

Key Insights and Ideas

  • Relationships over Raw Pixels: The most significant insight is that distilling relationships (via the Non-Local module) is more robust for continual learning than standard feature distillation. It helps the model retain the "logic" of how to distinguish objects without being overly rigid about specific feature activations.
  • The Power of Replay: Experience replay remains the most effective tool against catastrophic forgetting, providing a massive +14.1% boost in mAP compared to a model without it.
  • Detector Choice Matters: While many researchers focus on DETR-based architectures lately, the authors found that for this specific challenge, a refined one-stage dense detector (VFNet) outperformed more complex transformer-based models.
  • Simplicity in Sampling: Video-wise random sampling proved to be a simple yet effective way to manage the replay buffer in a video-based dataset.

Key Results

The nVFNet-RDC approach secured 1st place in both challenge tracks:

  • Track 2 (Category-level): Achieved 55.94% average mAP.
  • Track 3 (Instance-level): Achieved 54.65% average mAP.

Ablation studies showed the cumulative impact of their choices:

  • Replay: +14.1% mAP improvement.
  • COCO Pre-training: +4.08% mAP improvement.
  • Non-Local Distillation: +1.82% mAP improvement.
  • Res2Net Backbone: +1.70% mAP improvement over standard ResNet50.

Limitations and Future Work

The authors were transparent about several "Failed Attempts":

  • DETR/Deformable DETR: These popular transformer-based detectors were deemed too time-consuming for the challenge constraints (training time > 24h).
  • Advanced Augmentations: Techniques like Mosaic and Multi-scale data augmentation actually caused performance to drop, suggesting that egocentric data requires a more tailored approach to augmentation.
  • Long-tail Strategies: Standard class-balancing techniques were not effective here.
  • Multi-tasking: Adding scene or video classification branches to the detector also led to performance degradation.

Future work might involve finding ways to make transformer-based architectures more efficient for CL or developing augmentation strategies specifically tuned for first-person video data.

Summary updated

Abstract

Continual Learning (CL) focuses on developing algorithms with the ability to adapt to new environments and learn new skills. This very challenging task has generated a lot of interest in recent years, with new solutions appearing rapidly. In this paper, we propose a nVFNet-RDC approach for continual object detection. Our nVFNet-RDC consists of teacher-student models, and adopts replay and feature distillation strategies. As the 1st place solutions, we achieve 55.94% and 54.65% average mAP on the 3rd CLVision Challenge Track 2 and Track 3, respectively.
BibTeX
@misc{lai2022nvfnet,
  author = {Jinxiang Lai and Wenlong Liu and Jun Liu},
  title = {{nVFNet-RDC: Replay and Non-Local Distillation Collaboration for Continual Object Detection}},
  year = {2022},
  eprint = {2209.03603},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2209.03603},
}
Figure