Abstract
Object recognition ability is indispensable for robots to act like humans in a home environment. For example, when considering an object searching task, humans can recognize a naturally arranged object previously held in their hands while ignoring never observed objects. Even in such a simple task, we need to deal with three complex problems: domain adaptation, open-set recognition, and continual learning. However, most existing datasets are simplified to focus on one problem and do not measure the object recognition ability for home robots when multiple problems are simultaneously present. In this paper, we propose the COSDA-HR (Continual Open Set Domain Adaptation for Home Robot) dataset that requires dealing with the above three problems simultaneously. The COSDA-HR dataset focuses particularly on the scenario in which naturally arranged objects in a room are recognized by training with handheld objects towards the goal of creating a user-friendly teaching system for home robots. We provide various baselines to address the problems in the COSDA-HR dataset by combining state-of-the-art methods from each research area and analyze the limitations of such simple combinations. We consider that it is necessary to study the methods of handling multiple problems simultaneously instead of solving each problem to realize practical object recognition systems for home robots.
Introduction: The "Triple Threat" of Real-World AI
For a home-assistant robot to be truly useful, it must be able to learn from a user in a natural way. In a typical "user-friendly teaching" scenario, a human might hold an object up to the robot's camera to teach it. However, recognizing that same object later in a cluttered home environment presents three massive hurdles that are usually studied in isolation:
- Domain Adaptation (DA): Objects look significantly different when held in a hand (clean background, close-up) versus sitting on a cluttered, shadowy shelf (the target domain).
- Open-Set Recognition (OS): The robot's environment is filled with "unknown" objects. It must distinguish between known items and novel objects it should ignore.
- Continual Learning (CL): As users acquire new items, the robot must learn them without "catastrophically forgetting" previously learned objects.
This paper introduces the concept of Continual Open Set Domain Adaptation (COSDA), arguing that these problems must be solved simultaneously for practical home robotics.
The COSDA-HR Dataset
The researchers developed the COSDA-HR (Continual Open Set Domain Adaptation for Home Robot) dataset to benchmark this challenge.
- Source Data: Consists of 53,991 images of 160 object instances across 16 categories (e.g., bowls, bottles, toy blocks) held by a human hand in front of a Toyota HSR robot.
- Target Data: Captured during 22 robot explorations in a real home environment. This includes "known" objects placed naturally and "unknown" objects that serve as distractors.
- Environmental Challenges: The dataset includes significant lighting variations, comparing performance during the day (natural sunlight) versus at night (artificial indoor lighting).
Methodology
To evaluate current capabilities, the authors established a two-stage pipeline. First, an "objectness detector" (RetinaNet) identifies potential objects in the room. Second, a classification model identifies the object or rejects it as "unknown."
Since no single algorithm currently addresses all three problems, the researchers evaluated "stitched" combinations of state-of-the-art (SOTA) methods, such as combining Universal Domain Adaptation (UDA) with Bias Correction (BiC) for continual learning and OpenMax for open-set rejection. Performance was measured using a specialized top-1 (t=n) accuracy metric, which requires the model to maintain a specific rejection rate (e.g., 90%) for unknown objects to prevent "cheating" by misclassifying unknowns as knowns.
Key Results and Insights
The study's findings highlight the limitations of current modular AI approaches:
- The Failure of Simple Combinations: Simply plugging SOTA modules together is insufficient. A combination effective for DA and CL often fails when the Open-Set requirement is introduced.
- The Accuracy-Rejection Trade-off: There is a severe penalty for robustness. When required to reject 90% of unknown objects, the accuracy for known objects dropped to approximately 17%.
- Environmental Sensitivity: Performance plummeted at night, demonstrating that domain adaptation remains highly sensitive to lighting changes despite using SOTA techniques.
- Memory is Essential: Methods lacking a "rehearsal" mechanism (keeping a small memory of old images) suffered from catastrophic forgetting, making them unsuitable for long-term home use.
Conclusion
The COSDA-HR dataset reveals a significant gap between current AI capabilities and the requirements for home robotics. While modular solutions exist for individual problems, their combination creates new conflicts. Future work should focus on end-to-end systems that integrate detection and classification, as well as incorporating depth information and better uncertainty estimation to distinguish between known objects at difficult angles and truly novel items.
Summary updated