Wanderlust: Online Continual Object Detection in the Real World

Jianren Wang, Xin Wang, Yue Shang-Guan, Abhinav GuptaICCV 2021 · DOI

TL;DR

This paper introduces "Wanderlust," a new benchmark and dataset for online continual object detection in real-world, egocentric video. It addresses the critical gap in realistic evaluation for embodied AI agents that must learn continuously from dynamic, non-stationary data streams, unlike traditional offline methods.

What Problem Does This Paper Address?

Modern object detectors excel in controlled, offline settings where all data and categories are known beforehand. However, real-world embodied agents, like robots or even humans, operate differently. They encounter a continuous stream of visual information, where new object categories can appear at any time, and the data distribution naturally shifts as they move through different environments. Existing benchmarks often rely on static images or artificially split datasets, failing to capture the complexities of this "online continual learning" scenario. This leads to a significant challenge: how can AI agents learn new objects and adapt to changing environments without forgetting what they've already learned (catastrophic forgetting)?

Methods

The core contribution is the Wanderlust benchmark, built upon the Objects Around Krishna (OAK) dataset. OAK consists of egocentric videos captured by a graduate student over nine months, providing a realistic, long-term data stream. The benchmark simulates an online continual learning setting where the model trains and evaluates on sequential data.

Key methodological aspects include:

  • Egocentric Video Data: Using videos from a first-person perspective, mimicking how embodied agents perceive the world.
  • Natural Task Emergence: New object categories appear organically as the person moves through different locations and activities, rather than being artificially introduced.
  • Online Training/Evaluation: Models are trained and evaluated concurrently on the data stream, reflecting real-time learning.
  • New Evaluation Metrics: To capture the nuances of online continual learning, the authors introduce metrics like Continual Average Precision (CAP), Final Average Precision (FAP), Forward Transfer (FWT), Backward Transfer (BWT), and Forgetfulness (F). These metrics account for temporal dynamics and the impact of learning new tasks on previously learned ones.
  • "Unknown" Setting with IDK Class: A more realistic scenario where the model doesn't know the full vocabulary of classes beforehand. It must learn to identify unseen objects and predict a special "I Don't Know" (IDK) class.

Datasets

  • OAK (Objects Around Krishna) Dataset: A newly introduced dataset comprising 80 annotated video snippets (approx. 17.5 hours) from the KrishnaCam dataset. It features 105 object categories in outdoor scenes, with bounding box annotations at a 0.5 fps rate. The dataset is designed to reflect natural data distribution shifts and task emergence.
  • PASCAL VOC: Used for pre-training object detectors, providing a baseline for comparison.

Key Insights and Ideas

  • Realistic Benchmark is Crucial: Existing benchmarks are insufficient for evaluating embodied AI in dynamic, real-world scenarios. OAK and the Wanderlust benchmark provide a much-needed realistic testbed.
  • Egocentric Videos are Challenging: The inherent characteristics of egocentric videos (motion blur, occlusions, long-tail distributions, skewed box sizes) make object detection significantly harder than on static internet images.
  • Online Continual Learning is Difficult: Even standard continual learning algorithms struggle significantly on this benchmark, with CAP scores below 20, highlighting the need for novel approaches.
  • Forgetting is a Major Hurdle: The temporal nature of the data and the sequential learning process exacerbate catastrophic forgetting, even for sophisticated algorithms.
  • IDK Class is More Realistic: The "unknown" setting with an IDK class better reflects real-world scenarios where agents encounter novel objects.

Key Results

  • Pre-trained models perform poorly: A Faster R-CNN model pre-trained on PASCAL VOC achieved a low CAP of 2.86 on OAK, indicating a large domain gap.
  • Offline training is better but still limited: Even offline training on the OAK data achieved only 37.11 CAP, showing the inherent difficulty of the dataset.
  • Continual learning algorithms show modest gains: Among the tested algorithms (Incremental fine-tuning, iCaRL, EWC), iCaRL performed best with a CAP of 16.39 in the "known" setting, but still significantly lagged behind offline training.
  • Unknown setting is harder: In the "unknown" setting, iCaRL achieved a CAP of 14.70, further demonstrating the challenge of handling novel classes.
  • Forgetting is significant: Forgetfulness metrics showed that models do indeed forget previously learned information over time, though iCaRL was better at mitigating this than other baselines.

Limitations and Future Work

  • Limited Scope of Algorithms: The paper evaluates only a few representative continual learning algorithms. Future work could explore more advanced methods.
  • Domain Gap: The significant performance drop from PASCAL VOC to OAK highlights the need for methods that can better bridge domain gaps in continual learning.
  • Algorithm Specialization: The authors suggest that current continual learning algorithms, often designed for classification, may need to be specialized for object detection tasks.
  • Future Research: The paper calls for research into more robust algorithms for embodied online continual learning, particularly those that can handle the unique challenges of egocentric video and naturalistic data streams.

Summary updated

BibTeX
@inproceedings{wang2021wanderlust,
  author = {Jianren Wang and Xin Wang and Yue Shang-Guan and Abhinav Gupta},
  title = {{Wanderlust: Online Continual Object Detection in the Real World}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01065},
}
Figure