Sampling Training Data for Continual Learning Between Robots and the Cloud

Sandeep Chinchali, Evgenya Pergament, Manabu Nakanoya, Eyal Cidon, Edward Zhang, Dinesh Bharadia, Marco Pavone, Sachin Katti

TL;DR

Researchers have developed HARVESTNET, an intelligent sampling algorithm that allows robots to selectively "harvest" only the most useful training data from their massive sensory streams. By using cloud feedback to adaptively tune on-board sampling, HARVESTNET reduces cloud storage and annotation costs by up to 80% while significantly improving model accuracy compared to traditional methods.

What Problem Does This Paper Address?

We are entering the era of massive robotic fleets—think delivery drones and self-driving cars. These robots generate an astronomical amount of data; a single autonomous vehicle can produce upwards of 4 Terabytes of video and LIDAR data per day.

While this data is a goldmine for "continual learning" (improving models over time), it creates a massive systems bottleneck. Uploading everything to the cloud is impossible due to limited bandwidth, and even if you could, the costs for cloud storage, human annotation (labeling images), and compute time would be ruinous. The challenge is: How does a robot decide, in real-time, which few frames are actually worth saving and sending to the cloud to improve its brain?

Methods

The authors introduce HARVESTNET, a compute-efficient sampler designed to run on resource-constrained edge hardware (like a Raspberry Pi with a Google Edge TPU).

The system works in a loop:

  1. On-board Inference: As the robot moves, its vision model processes frames. HARVESTNET looks at the "embeddings" (mathematical fingerprints of images) and "softmax confidences" (how sure the model is about what it sees).
  2. Intelligent Sampling: It uses two main strategies to pick images:
  3. Embedding Distance: It compares new images to a small "target set" of interesting images provided by the cloud. If the new image is "close" enough in fingerprint space, it’s saved.
  4. Confidence Thresholding: It identifies images where the model sees a target class (like a construction site) but needs more examples to refine its understanding.
  5. Cloud Feedback (The "Aha!" Moment): This is the novel part. Instead of the robot just sending data one way, the cloud periodically re-trains the model and sends back updated sampling thresholds. This "closes the loop," allowing the robot to become a more surgical data collector over time.

Datasets

The study evaluated HARVESTNET across three distinct domains:

  • Identifying Construction Sites: A novel dataset collected over three months (346,788 frames) featuring rare, safety-critical scenarios like road excavations and "compactor" machines.
  • Self-Driving Vehicles (Waymo): Using the Waymo Open Dataset to identify rare sightings of other autonomous vehicles on public roads.
  • Streaming Face Recognition (FaceNet): A search-and-rescue scenario where the robot must identify specific faces from a video stream.

Key Insights and Ideas

  • Transfer Learning is Key: You don't need millions of images to specialize a model. By freezing the early layers of a neural network (which act as general feature extractors) and only fine-tuning the final layers, robots can achieve high accuracy with very few sampled images.
  • The Power of the Loop: Most samplers are "static." HARVESTNET is "adaptive." By letting the cloud tell the robot, "Hey, I'm getting enough of Category A, start looking harder for Category B," the system avoids redundant data.
  • Embeddings are "Free" Metadata: Since the robot is already running a vision model for navigation or planning, the embeddings and confidence scores are generated anyway. HARVESTNET "harvests" this data for sampling decisions at almost zero extra computational cost (~0.3ms per frame).

Key Results

  • Cost Savings: HARVESTNET reduced cloud storage, human annotation time, and cloud compute time by 65.7% to 81.3%.
  • Accuracy Gains: Despite sending far less data, the models re-trained via HARVESTNET were 1.05x to 2.58x more accurate than those using random sampling or non-adaptive baselines.
  • Oracle Performance: In some tests (like the FaceNet dataset), HARVESTNET's performance nearly matched an "Oracle"—an impossible theoretical sampler that perfectly identifies every target image.
  • Edge Ready: The researchers demonstrated the system running on a Raspberry Pi and Google Edge TPU, proving it can handle real-time video streams on low-power robots.

Limitations and Future Work

  • Known vs. Unknown Weakpoints: Currently, HARVESTNET excels at finding more examples of things we know are problems (e.g., "we need more construction sites"). It is less optimized for finding "unknown unknowns"—completely new visual concepts the developers haven't anticipated.
  • Sensor Fusion: The current work focuses heavily on vision. The authors suggest that future versions could fuse data from multiple cameras and LIDAR sensors simultaneously to make even better sampling decisions.
  • Privacy: While the paper touches on privacy in the FaceNet section, the authors note that future work could explicitly modify the sampler to reject private data, ensuring sensitive information never leaves the robot.

Summary updated

Abstract

Today’s robotic fleets are increasingly measuring high-volume video and LIDAR sensory streams, which can be mined for valuable training data to continually improve robotic perception models. However, re-training perception models on growing volumes of rich sensory data in central servers (or the “cloud”) incurs significant systems costs for network transfer, cloud storage, human annotation, and cloud computing resources. Hence, we introduce HarvestNet, an intelligent sampling algorithm that resides on-board a robot and reduces system bottlenecks by only storing rare, useful events to continually improve and re-train perception models. HarvestNet significantly improves the accuracy of perception models on our novel dataset of road construction sites, field testing of self-driving cars, and face recognition, while reducing cloud storage, dataset annotation time, and cloud compute time by between 65.7−81.3 $$\%$$ . Further, it is between 1.05−2.58 $$\times $$ more accurate than baseline algorithms and runs on low-power deep learning hardware.
BibTeX
@inproceedings{chinchali2020sampling,
  author = {Sandeep Chinchali and Evgenya Pergament and Manabu Nakanoya and Eyal Cidon and Edward Zhang and Dinesh Bharadia and Marco Pavone and Sachin Katti},
  title = {{Sampling Training Data for Continual Learning Between Robots and the Cloud}},
  booktitle = {ISER},
  year = {2020},
  url = {https://openreview.net/forum?id=uhkRmGsSuQ3},
}

Loading related content...

Figure