Fast Object Segmentation Learning with Kernel-based Methods for Robotics

Federico Ceola, Elisa Maiettini, Giulia Pasquale, Lorenzo Rosasco, Lorenzo Natale

Abstract

Object segmentation is a key component in the visual system of a robot that performs tasks like grasping and object manipulation, especially in presence of occlusions. Like many other computer vision tasks, the adoption of deep architectures has made available algorithms that perform this task with remarkable performance. However, adoption of such algorithms in robotics is hampered by the fact that training requires large amount of computing time and it cannot be performed on-line. In this work, we propose a novel architecture for object segmentation, that overcomes this problem and provides comparable performance in a fraction of the time required by the state-of-the-art methods. Our approach is based on a pre-trained Mask R-CNN, in which various layers have been replaced with a set of classifiers and regressors that are retrained for a new task. We employ an efficient Kernel-based method that allows for fast training on large scale problems. Our approach is validated on the YCB-Video dataset which is widely adopted in the computer vision and robotics community, demonstrating that we can achieve and even surpass performance of the state-of-the-art, with a significant reduction (~6×) of the training time.

Introduction

In robotics, a "one-size-fits-all" vision model rarely suffices. Robots frequently encounter novel objects in specific environments and must learn to recognize and segment them "on-line"—quickly and on-site. While deep learning architectures like Mask R-CNN provide excellent segmentation performance, their training is computationally intensive. Standard fine-tuning requires expensive backpropagation and extensive hyperparameter tuning, often taking hours even on high-end GPUs. This computational burden prevents robots from learning new objects during interactive sessions. This paper proposes a hybrid architecture that bridges the gap between high-performance deep learning and the agility required for real-world robotics.

Proposed Architecture

The core innovation is a "decoupled" approach that separates general feature extraction from task-specific learning. Instead of training a monolithic network end-to-end, the architecture is split into two modules:

  1. Feature Extraction Module: A backbone and Region Proposal Network (RPN) from a Mask R-CNN pre-trained on a general dataset (e.g., MS COCO). These layers act as a fixed feature extractor for shapes, edges, and textures.
  2. On-line Learning Modules: The standard fully-connected "heads" are replaced with FALKON, a state-of-the-art Nyström-based kernel method.
  3. Detection Module: Employs FALKON classifiers for object identification and Regularized Least Squares (RLS) for bounding box refinement.
  4. Segmentation Module: Uses per-pixel FALKON classifiers for each class to learn "figure-ground" masks rapidly.
Figure 1: The proposed hybrid architecture decoupling deep feature extraction from fast kernel-based on-line detection and segmentation modules.

To maintain efficiency with massive pixel data, the authors utilize a minibootstrap technique for detection and data subsampling for segmentation. Subsampling up to 90% of training pixels was found to have minimal impact on accuracy while drastically reducing training time.

Experimental Results

The approach was validated using the YCB-Video dataset, a standard robotics benchmark featuring 21 objects in cluttered and occluded environments. The model's general features were initialized via pre-training on MS COCO.

The hybrid approach was compared against a standard Mask R-CNN baseline where only the output layers were fine-tuned. Key findings include:

  • Training Speed: The proposed method achieved a ~6x speedup, training in approximately 19.5 minutes compared to the baseline's 2 hours. The mask-learning component specifically required only 34 seconds.
  • Segmentation Accuracy: Despite the speedup, the model maintained high performance, achieving a segmentation mAP (at 50% IoU) of 74.66% on YCB-Video, slightly exceeding the baseline's 74.12%.
  • Robustness: The system demonstrated strong performance under heavy occlusion, a critical requirement for robotic manipulation.
Figure 5: Qualitative results showing predicted masks and detections on objects from the YCB-Video dataset.

Conclusion

By replacing the final layers of a deep network with efficient kernel-based classifiers, this work enables fast, on-line learning for robotic vision. While the method depends on the quality of the pre-trained backbone, it significantly reduces the computational barrier to deploying advanced segmentation in dynamic environments. Future work includes extending this pipeline to 6D pose estimation and integrating few-shot learning to further minimize data requirements.

Summary updated

Abstract

Object segmentation is a key component in the visual system of a robot that performs tasks like grasping and object manipulation, especially in presence of occlusions. Like many other computer vision tasks, the adoption of deep architectures has made available algorithms that perform this task with remarkable performance. However, adoption of such algorithms in robotics is hampered by the fact that training requires large amount of computing time and it cannot be performed on-line. In this work, we propose a novel architecture for object segmentation, that overcomes this problem and provides comparable performance in a fraction of the time required by the state-of-the-art methods. Our approach is based on a pre-trained Mask R-CNN, in which various layers have been replaced with a set of classifiers and regressors that are re-trained for a new task. We employ an efficient Kernel-based method that allows for fast training on large scale problems. Our approach is validated on the YCB-Video dataset which is widely adopted in the computer vision and robotics community, demonstrating that we can achieve and even surpass performance of the state-of-the-art, with a significant reduction (${\sim}6\times$) of the training time. The code to reproduce the experiments is publicly available on GitHub.
BibTeX
@inproceedings{ceola2021fast,
  author = {Federico Ceola and Elisa Maiettini and Giulia Pasquale and Lorenzo Rosasco and Lorenzo Natale},
  title = {{Fast Object Segmentation Learning with Kernel-based Methods for Robotics}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=2uWBC-VmYo},
}
Figure