Abstract
Today's robotic fleets are increasingly measuring high-volume video and LIDAR sensory streams, which can be mined for valuable training data, such as rare scenes of road construction sites, to steadily improve robotic perception models. However, re-training perception models on growing volumes of rich sensory data in central compute servers (or the "cloud") places an enormous time and cost burden on network transfer, cloud storage, human annotation, and cloud computing resources. Hence, we introduce HarvestNet, an intelligent sampling algorithm that resides on-board a robot and reduces system bottlenecks by only storing rare, useful events to steadily improve perception models re-trained in the cloud. HarvestNet significantly improves the accuracy of machine-learning models on our novel dataset of road construction sites, field testing of self-driving cars, and streaming face recognition, while reducing cloud storage, dataset annotation time, and cloud compute time by between 65.7-81.3%. Further, it is between 1.05-2.58x more accurate than baseline algorithms and scalably runs on embedded deep learning hardware. We provide a suite of compute-efficient perception models for the Google Edge Tensor Processing Unit (TPU), an extended technical report, and a novel video dataset to the research community at https://sites.google.com/view/harvestnet.
Introduction
As robotic fleets like autonomous vehicles and delivery drones become more common, they generate an astronomical amount of data—upwards of 4 Terabytes per day for a single vehicle. While this data is essential for "continual learning" to improve robotic brains over time, the sheer volume creates a massive systems bottleneck. Uploading every frame to the cloud is prohibited by limited bandwidth, and the subsequent costs for storage, human annotation, and compute time are unsustainable. The core challenge is enabling a robot to decide, in real-time, which specific frames are worth saving and transmitting to improve its perception models.
The HarvestNet System
We introduce HarvestNet, a compute-efficient sampler designed to run on resource-constrained edge hardware, such as a Raspberry Pi equipped with a Google Edge TPU. HarvestNet operates in a closed-loop cycle between the robot and the cloud to optimize data collection.
Intelligent On-Board Sampling
HarvestNet utilizes two primary strategies to identify "interesting" images:
- Embedding Distance: It compares the mathematical "fingerprints" (embeddings) of new images to a target set of interesting examples provided by the cloud. If a new image is sufficiently similar to these targets, it is cached for upload.
- Confidence Thresholding: It identifies images where the model detects a target class (e.g., a construction site) but exhibits low softmax confidence, indicating a need for more training data to refine its understanding.
Cloud-to-Edge Feedback
Unlike static samplers, HarvestNet is adaptive. The cloud periodically re-trains the perception model and sends updated sampling thresholds back to the robot. This feedback allows the robot to become a more surgical collector, avoiding redundant data and focusing on current model weaknesses.
Experimental Evaluation
HarvestNet was evaluated across three distinct domains:
- Road Construction Sites: A novel dataset featuring rare, safety-critical scenarios like excavations and specialized machinery.
- Self-Driving Vehicles: Identifying rare sightings of other autonomous vehicles using the Waymo Open Dataset.
- Streaming Face Recognition: A search-and-rescue scenario requiring the identification of specific faces from a video stream.
Key Results
- Efficiency: HarvestNet reduced cloud storage, human annotation time, and cloud compute time by 65.7% to 81.3%.
- Accuracy: Despite the massive reduction in data volume, models re-trained via HarvestNet were 1.05x to 2.58x more accurate than those using random sampling or non-adaptive baselines.
- Edge Performance: The system runs with minimal overhead (~0.3ms per frame), making it ideal for real-time deployment on low-power robotic hardware.
Conclusion
HarvestNet demonstrates that intelligent, adaptive sampling can overcome the data bottlenecks of large-scale robotic fleets. By closing the loop between edge perception and cloud training, robots can learn more effectively while significantly reducing the infrastructure costs associated with continual learning.
Summary updated