How Robots Learn to Classify New Objects Trained from Small Data Sets

Tick Son Wang, Zoltan-Csaba Marton, Manuel Brucker, Rudolph TriebelCoRL 2017 · OpenReview

Abstract

In this paper, we address the problem of learning to classify new object classes and instances by adapting a previously trained classifier. The main challenges here are the small amount of newly available training data and the large change in appearance between the new and the old data. To address this we propose a new variant of Progressive Neural Networks (PNN), originally introduced by Rusu et al. We show that by performing a specific simplification in the adapters, the prediction performance of the resulting PNN can be significantly increased. Furthermore, we give additional insights about when PNNs outperform alternative methods, and provide empirical evaluations on benchmark datasets. Finally, we also suggest a way of using it to augment the functionality of a network by extending it with new classes, addressing the problem of unbalanced classes, i.e. where the new classes are under-represented.

Introduction

Deep learning models are notoriously data-hungry, yet robots in real-world environments often encounter a "small data" problem. When a robot is introduced to a new object or environment, it may only have a handful of images to learn from. The standard approach of fine-tuning a pre-trained model often suffers from two major flaws: catastrophic forgetting of original tasks and poor performance when there is a significant domain mismatch (e.g., moving from color RGB images to Depth maps).

This paper proposes an improved Progressive Neural Network (PNN) architecture designed for lifelong learning, allowing robots to expand their knowledge base without losing prior capabilities, even when training data is sparse.

Methods: Improving Progressive Neural Networks

The proposed architecture builds upon the PNN framework, where a frozen "source" network is augmented with a new "target" network column. Lateral connections, known as adapters, allow the target network to leverage features from the source. The authors introduce two key technical innovations:

  1. Simplified Adapters: Original PNN designs utilized complex, multi-layered adapters. The authors found that for robotic vision tasks with limited data, these complex structures often led to overfitting. They proposed a simplified "Project-adapter"—a 1x1 convolution followed by a ReLU activation—which acts as a more efficient dimensionality-matching bridge.
  2. The Relay Mechanism: To handle the addition of new classes, the authors introduced a "relay class." In this "Extended Classifier" setup, if the new network is uncertain about a classification, it triggers a relay signal to the original source network. This ensures the system can handle both old and new objects seamlessly.
The proposed PNN architecture (left) utilizes simplified 1x1 convolutional adapters (V) compared to the original PNN design (right).

Datasets and Evaluation

The approach was evaluated using standard robotics benchmarks, including the Washington University RGB-D Object Dataset (WRGBD), YCB Object Set, and BigBIRD. By transferring knowledge between these datasets (e.g., from BigBIRD high-quality scans to WRGBD robotic captures), the researchers simulated the domain shifts robots experience in the wild.

Key Results and Insights

The proposed PNN demonstrated significant robustness in "small data" scenarios. While fine-tuning accuracy dropped by approximately 20% when training data was reduced, the PNN approach only saw a 3% decrease.

Performance comparison of fine-tuning versus the proposed PNN across RGB (left) and Depth (right) modalities for the WRGBD-Replica dataset.

Key findings include:

  • Less is More: Simplifying the adapter architecture significantly increased performance on sparse data by reducing overfitting.
  • Hard Transfer Excellence: PNNs are particularly powerful when source and target data differ significantly (e.g., RGB to Depth). The frozen source network provides a stable foundation that fine-tuning might otherwise scramble.
  • Effective Expansion: On the BigBIRD to WRGBD transfer task, the proposed PNN achieved 74.71% accuracy, outperforming both the original PNN (69.72%) and standard fine-tuning (71.38%).

Conclusion

While PNNs face scalability challenges due to the addition of new network columns for each task, they offer a superior path for lifelong robotic learning compared to traditional fine-tuning. Future work will focus on applying these techniques to more complex tasks like object detection and semantic segmentation, as well as exploring automated column pruning to manage computational costs.

Summary updated

Abstract

In this paper, we address the problem of learning to classify new object classes and instances by adapting a previously trained classifier. The main challenges here are the small amount of newly available training data and the large change in appearance between the new and the old data. To address this we propose a new variant of Progressive Neural Networks (PNN), originally introduced by Rusu et al. We show that by performing a specific simplification in the adapters, the prediction performance of the resulting PNN can be significantly increased. Furthermore, we give additional insights about when PNNs outperform alternative methods, and provide empirical evaluations on benchmark datasets. Finally, we also suggests a way of using it to augment the functionality of a network by extending it with new classes, addressing the problem of unbalanced classes, i.e. where the new classes are under-represented.
BibTeX
@inproceedings{wang2017how,
  author = {Tick Son Wang and Zoltan-Csaba Marton and Manuel Brucker and Rudolph Triebel},
  title = {{How Robots Learn to Classify New Objects Trained from Small Data Sets}},
  booktitle = {Conference on Robot Learning},
  year = {2017},
  url = {https://openreview.net/forum?id=NoI5DaKp69},
}
Figure