ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning

Ryan Hoque, Ashwin Balakrishna, Ellen R. Novoseller, Albert Wilcox, Daniel S. Brown, Ken Goldberg

Introduction

Imitation Learning (IL) is a powerful paradigm for teaching robots complex tasks through human demonstrations. However, offline IL often fails due to "distribution shift," where small errors lead the robot into unfamiliar states from which it cannot recover. Interactive IL (e.g., DAgger) addresses this by allowing online human corrections. Yet, constant human monitoring is impractical for large-scale deployments or robot fleets. ThriftyDAgger addresses this by introducing a budget-aware, dual-gating mechanism that intelligently requests human intervention only when necessary.

Methodology: Dual-Gating and Risk Estimation

ThriftyDAgger's core innovation is a dual-gating system that decides when to cede control to a human supervisor based on two metrics: Novelty and Risk.

ThriftyDAgger Architecture: The system uses a dual-gating mechanism (Novelty and Risk) to decide when to request human intervention or proceed autonomously based on a user-defined budget.
  1. Novelty Estimation: The robot employs an ensemble of policies. Significant disagreement among ensemble members indicates a novel state, triggering a request for help.
  2. Risk Estimation: Unlike prior methods that rely on action discrepancy, ThriftyDAgger trains a specialized Q-function to estimate the discounted probability of task success. Risk is defined as the complement of this success probability. This allows the robot to identify "bottlenecks"—states that might not be novel but where errors are catastrophic.
  3. Budget-Aware Gating: Users provide a fixed intervention budget ($\alpha_h$). The system automatically calibrates novelty and risk thresholds using the quantiles of visited states to respect this budget.
  4. Hysteresis Control: To ensure smooth transitions and prevent rapid switching (chattering), the algorithm applies stricter criteria for ending an intervention than for starting one.

Experimental Evaluation

The authors evaluated ThriftyDAgger across simulated and physical domains, focusing on tasks with high precision requirements and complex contact dynamics.

Task Visualizations: Examples of Start, Goal, Novel, Risky, and Safe states for the Peg Insertion (Simulation) and Cable Routing (Physical) tasks.
  • Peg Insertion (Simulation): A task requiring grasping a ring and threading it onto a cylinder.
  • Physical Cable Routing: Performed on a da Vinci surgical robot, where the system must route a cable through pegs in a figure-8 pattern using raw image input.
  • Fleet Management User Study: 10 participants managed three robots simultaneously while performing a secondary cognitive task (a memory game) to simulate real-world distractions.

Key Results and Insights

ThriftyDAgger significantly outperformed existing "robot-gated" baselines like SafeDAgger and LazyDAgger.

  • Performance and Autonomy: In the Peg Insertion task, ThriftyDAgger achieved a 73% autonomous success rate, nearly triple that of SafeDAgger (24%). On the physical da Vinci robot, it achieved a 100% intervention-aided success rate.
  • Human-Robot Synergy: The user study revealed that ThriftyDAgger increased robot throughput by 80% compared to the next best algorithm. Interestingly, the robot-gated system outperformed human-gated systems, suggesting that robots are better at identifying their own subtle instabilities than humans are at monitoring multiple concurrent tasks.
  • Reduced Supervisor Burden: Participants reported lower mental load and frustration, as the system effectively managed the "monitoring bottleneck."

Conclusion

ThriftyDAgger demonstrates that explicitly modeling success probability (Risk) and respecting a human intervention budget (Novelty) creates a more robust and practical interactive learning system. By shifting from arbitrary thresholds to resource-aware gating, it paves the way for efficient human-supervised robot fleets. Future work will explore time-varying budgets and applications in Human-in-the-Loop Reinforcement Learning.

Summary updated

BibTeX
@inproceedings{hoque2021thriftydagger,
  author = {Ryan Hoque and Ashwin Balakrishna and Ellen R. Novoseller and Albert Wilcox and Daniel S. Brown and Ken Goldberg},
  title = {{ThriftyDAgger: Budget-Aware Novelty and Risk Gating for Interactive Imitation Learning}},
  booktitle = {CoRL},
  year = {2021},
  url = {https://openreview.net/forum?id=KKBfrCzCVOn},
}
Figure