Sampling Training Data for Continual Learning Between Robots and the Cloud

Sandeep Chinchali, Evgenya Pergament, Manabu Nakanoya, Eyal Cidon, Edward Zhang, Dinesh Bharadia, Marco Pavone, Sachin Katti

Abstract

Today's robotic fleets are increasingly measuring high-volume video and LIDAR sensory streams, which can be mined for valuable training data, such as rare scenes of road construction sites, to steadily improve robotic perception models. However, re-training perception models on growing volumes of rich sensory data in central compute servers (or the "cloud") places an enormous time and cost burden on network transfer, cloud storage, human annotation, and cloud computing resources. Hence, we introduce HarvestNet, an intelligent sampling algorithm that resides on-board a robot and reduces system bottlenecks by only storing rare, useful events to steadily improve perception models re-trained in the cloud. HarvestNet significantly improves the accuracy of machine-learning models on our novel dataset of road construction sites, field testing of self-driving cars, and streaming face recognition, while reducing cloud storage, dataset annotation time, and cloud compute time by between 65.7-81.3%. Further, it is between 1.05-2.58x more accurate than baseline algorithms and scalably runs on embedded deep learning hardware. We provide a suite of compute-efficient perception models for the Google Edge Tensor Processing Unit (TPU), an extended technical report, and a novel video dataset to the research community at https://sites.google.com/view/harvestnet.

Introduction

As robotic fleets like autonomous vehicles and delivery drones become more common, they generate an astronomical amount of data—upwards of 4 Terabytes per day for a single vehicle. While this data is essential for "continual learning" to improve robotic brains over time, the sheer volume creates a massive systems bottleneck. Uploading every frame to the cloud is prohibited by limited bandwidth, and the subsequent costs for storage, human annotation, and compute time are unsustainable. The core challenge is enabling a robot to decide, in real-time, which specific frames are worth saving and transmitting to improve its perception models.

The HarvestNet System

We introduce HarvestNet, a compute-efficient sampler designed to run on resource-constrained edge hardware, such as a Raspberry Pi equipped with a Google Edge TPU. HarvestNet operates in a closed-loop cycle between the robot and the cloud to optimize data collection.

HarvestNet system architecture showing the closed-loop interaction between on-board sampling and cloud-based model re-training.

Intelligent On-Board Sampling

HarvestNet utilizes two primary strategies to identify "interesting" images:

  1. Embedding Distance: It compares the mathematical "fingerprints" (embeddings) of new images to a target set of interesting examples provided by the cloud. If a new image is sufficiently similar to these targets, it is cached for upload.
  2. Confidence Thresholding: It identifies images where the model detects a target class (e.g., a construction site) but exhibits low softmax confidence, indicating a need for more training data to refine its understanding.

Cloud-to-Edge Feedback

Unlike static samplers, HarvestNet is adaptive. The cloud periodically re-trains the perception model and sends updated sampling thresholds back to the robot. This feedback allows the robot to become a more surgical collector, avoiding redundant data and focusing on current model weaknesses.

Experimental Evaluation

HarvestNet was evaluated across three distinct domains:

  • Road Construction Sites: A novel dataset featuring rare, safety-critical scenarios like excavations and specialized machinery.
  • Self-Driving Vehicles: Identifying rare sightings of other autonomous vehicles using the Waymo Open Dataset.
  • Streaming Face Recognition: A search-and-rescue scenario requiring the identification of specific faces from a video stream.
An example of a rare road construction site from the novel dataset, representing the type of safety-critical data HarvestNet is designed to harvest.

Key Results

  • Efficiency: HarvestNet reduced cloud storage, human annotation time, and cloud compute time by 65.7% to 81.3%.
  • Accuracy: Despite the massive reduction in data volume, models re-trained via HarvestNet were 1.05x to 2.58x more accurate than those using random sampling or non-adaptive baselines.
  • Edge Performance: The system runs with minimal overhead (~0.3ms per frame), making it ideal for real-time deployment on low-power robotic hardware.

Conclusion

HarvestNet demonstrates that intelligent, adaptive sampling can overcome the data bottlenecks of large-scale robotic fleets. By closing the loop between edge perception and cloud training, robots can learn more effectively while significantly reducing the infrastructure costs associated with continual learning.

Summary updated

Abstract

Today's robotic fleets are increasingly measuring high-volume video and LIDAR sensory streams, which can be mined for valuable training data, such as rare scenes of road construction sites, to steadily improve robotic perception models. However, re-training perception models on growing volumes of rich sensory data in central compute servers (or the "cloud") places an enormous time and cost burden on network transfer, cloud storage, human annotation, and cloud computing resources. Hence, we introduce HarvestNet, an intelligent sampling algorithm that resides on-board a robot and reduces system bottlenecks by only storing rare, useful events to steadily improve perception models re-trained in the cloud. HarvestNet significantly improves the accuracy of machine-learning models on our novel dataset of road construction sites, field testing of self-driving cars, and streaming face recognition, while reducing cloud storage, dataset annotation time, and cloud compute time by between 65.7-81.3%. Further, it is between 1.05-2.58x more accurate than baseline algorithms and scalably runs on embedded deep learning hardware. We provide a suite of compute-efficient perception models for the Google Edge Tensor Processing Unit (TPU), an extended technical report, and a novel video dataset to the research community at https://sites.google.com/view/harvestnet.
BibTeX
@inproceedings{chinchali2020sampling,
  author = {Sandeep Chinchali and Evgenya Pergament and Manabu Nakanoya and Eyal Cidon and Edward Zhang and Dinesh Bharadia and Marco Pavone and Sachin Katti},
  title = {{Sampling Training Data for Continual Learning Between Robots and the Cloud}},
  booktitle = {ISER},
  year = {2020},
  url = {https://openreview.net/forum?id=uhkRmGsSuQ3},
}
Figure