Speeding-up Object Detection Training for Robotics with FALKON

Elisa Maiettini, Giulia Pasquale, Lorenzo Rosasco, Lorenzo Natale

Abstract

Latest deep learning methods for object detection provide remarkable performance, but have limits when used in robotic applications. One of the most relevant issues is the long training time, which is due to the large size and imbalance of the associated training sets, characterized by few positive and a large number of negative examples (i.e. background). Proposed approaches are based on end-to-end learning by back-propagation or kernel methods trained with Hard Negatives Mining on top of deep features. These solutions are effective, but prohibitively slow for on-line applications. In this paper we propose a novel pipeline for object detection that overcomes this problem and provides comparable performance, with a 60x training speedup. Our pipeline combines (i) the Region Proposal Network and the deep feature extractor from Faster R-CNN to efficiently select candidate RoIs and encode them into powerful representations, with (ii) the FALKON algorithm, a novel kernel-based method that allows fast training on large scale problems (millions of points). We address the size and imbalance of training data by exploiting the stochastic subsampling intrinsic into the method and a novel, fast, bootstrapping approach. We assess the effectiveness of the approach on a standard Computer Vision dataset (PASCAL VOC 2007) and demonstrate its applicability to a real robotic scenario with the iCubWorld Transformations dataset.

Introduction

In robotics, the ability to adapt quickly to new environments is vital. However, state-of-the-art object detection models like Faster R-CNN are notoriously slow to train. This "training bottleneck" stems from computational intensity—requiring massive GPU resources and long durations for back-propagation—and extreme data imbalance. In object detection, "background" examples (negatives) vastly outnumber actual objects (positives). Traditional methods use "Hard Negative Mining" to find the most informative background samples, but this process is iterative and extremely time-consuming.

For a robot interacting with a human in real-time, waiting hours for a model to update is impractical. This research seeks a way to perform "on-line" learning—training the final classification stage of a detector in seconds or minutes rather than hours.

Methodology

The proposed hybrid pipeline splits the detection task into off-line and on-line components to maximize efficiency:

  • Off-line (Fixed): The system utilizes a pre-trained Region Proposal Network (RPN) and a CNN feature extractor from the Faster R-CNN architecture. These layers are frozen, as deep features are often general-purpose enough to be reused for new, unseen objects.
  • On-line (Trainable): The standard fully-connected layers are replaced with FALKON, a state-of-the-art kernel-based method designed for large-scale datasets. FALKON employs Nyström subsampling (stochastically selecting a subset of data points as "kernel centers") and a preconditioned conjugate gradient solver to handle millions of data points efficiently.
  • Approximated Hard Negatives Mining: To solve the data imbalance problem quickly, a "mini-bootstrap" approach is introduced. Instead of checking every background patch in every image, the system samples random subsets and performs a fast, approximated version of hard negative selection.
  • Bounding Box Refinement: Regularized Least Squares (RLS) regressors are used to fine-tune the coordinates of the detected boxes.
The proposed pipeline integrates FALKON and RLS into the Faster R-CNN architecture for fast classification and bounding box refinement.

Key Results

The FALKON-based pipeline demonstrates a significant breakthrough for robotic adaptation:

  • 60x Speedup: On the iCubWorld dataset, fine-tuning a standard Faster R-CNN took approximately 40 minutes. The FALKON + Mini Bootstrap approach achieved comparable accuracy in just 40 to 50 seconds.
  • Accuracy Gains: On the iCubWorld Transformations (ICWT) target task, the proposed method reached 51.3% mAP, outperforming the Faster R-CNN baseline (49.7% mAP) while being significantly faster.
  • Scalability: As shown in the performance metrics, increasing the number of Nyström centers improves accuracy (mAP) while training and test times scale predictably. On PASCAL VOC, the model could be trained in just 25 seconds with only a 4% drop in performance compared to a full 25-minute deep learning training session.
Performance metrics (mAP, train time, test time) as a function of the number of Nystrom centers.

Conclusion

By decoupling feature extraction from classification and utilizing optimized kernel methods, this pipeline enables robots to learn new objects on-the-fly. While the system relies on the quality of pre-trained features and must manage the memory intensity of kernel methods, it provides a viable path toward continuous learning loops in autonomous systems. Future work will explore integrating this fast training into autonomous re-training cycles based on environmental confidence.

Summary updated

Abstract

Latest deep learning methods for object detection provide remarkable performance, but have limits when used in robotic applications. One of the most relevant issues is the long training time, which is due to the large size and imbalance of the associated training sets, characterized by few positive and a large number of negative examples (i.e. background). Proposed approaches are based on end-to-end learning by back-propagation [22] or kernel methods trained with Hard Negatives Mining on top of deep features [8]. These solutions are effective, but prohibitively slow for on-line applications. In this paper we propose a novel pipeline for object detection that overcomes this problem and provides comparable performance, with a 60x training speedup. Our pipeline combines (i) the Region Proposal Network and the deep feature extractor from [22] to efficiently select candidate RoIs and encode them into powerful representations, with (ii) the FALKON [23] algorithm, a novel kernel-based method that allows fast training on large scale problems (millions of points). We address the size and imbalance of training data by exploiting the stochastic subsampling intrinsic into the method and a novel, fast, bootstrapping approach. We assess the effectiveness of the approach on a standard Computer Vision dataset (PASCAL VOC 2007 [5]) and demonstrate its applicability to a real robotic scenario with the iCubWorld Transformations [18] dataset.
BibTeX
@inproceedings{maiettini2018speeding,
  author = {Elisa Maiettini and Giulia Pasquale and Lorenzo Rosasco and Lorenzo Natale},
  title = {{Speeding-up Object Detection Training for Robotics with FALKON}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  url = {https://openreview.net/forum?id=yK2C9VYgT-w},
}
Figure