Abstract
Reinforcement learning agents often fail during "sim-to-real" transfer because they encounter "blind spots"—regions where their simplified internal representation cannot distinguish between safe and dangerous real-world states. This paper introduces a framework that uses limited human feedback to train a predictive model of these blind spots, allowing agents to proactively "ask for help" before making a catastrophic mistake.
Introduction: The Sim-to-Real Gap
The "sim-to-real" gap is a notorious hurdle in robotics and autonomous systems. While training in simulation is cost-effective and safe, simulators are inevitably simpler than reality. This research focuses on a specific type of failure: representational incompleteness.
Imagine an autonomous car trained in a simulator that has never seen an ambulance. In the real world, the agent might perceive an ambulance simply as a "large white vehicle." Because its internal model lacks features like "siren" or "emergency lights," it treats the ambulance exactly like a standard delivery truck. If the agent continues driving normally, it makes a high-cost error. The agent isn't just wrong; it’s confident and wrong because it literally cannot see the features that necessitate a different action.
Methodology
The authors formalize blind spot detection as a noisy supervised learning problem with class imbalance. The proposed pipeline consists of three main stages:
- Data Collection via Oracles: The agent interacts with an "oracle" (a human or expert system) in the real world. Feedback is gathered through Corrections (the oracle monitors the agent and intervenes) or Demonstrations (the oracle shows the agent how to perform the task).
- Label Aggregation (Dawid-Skene): Due to the representational gap, one agent observation might correspond to multiple real-world states—some safe, some not. This creates "noise" in the labels. The authors use the Dawid-Skene algorithm (an Expectation-Maximization approach) to estimate label reliability and aggregate them into a "ground truth" probability that a specific observation is a blind spot.
- Model Learning and Calibration: A Random Forest classifier is trained to generalize from visited states to unseen regions of the state space. Since blind spots are rare, the researchers employ oversampling and temperature calibration to ensure the model’s probability estimates are useful for decision-making.
Finally, the system implements Oracle-in-the-loop (OIL) execution: during deployment, if the model predicts a high probability of a blind spot, the agent pauses and queries the oracle for the correct action.
Experimental Evaluation
The approach was evaluated on two modified classic environments designed to simulate representational gaps:
- Modified Catcher: The agent must catch falling fruit but lacks the features to distinguish between "good" fruit and "bad" (dangerous) fruit that appears identical in its representation.
- Modified Flappy Bird: The agent must fly between pipes. Some pipes are made of copper (creating dangerous wind conditions if the bird flies high), but the agent cannot "see" the pipe material.
Key Insights
- The Suboptimality Paradox: A surprising finding is that suboptimal oracles and simulator policies actually improve the model. If an oracle is "too perfect," it avoids dangerous regions entirely, leaving the agent with no data about its blind spots. Suboptimal behavior provides a more diverse set of labels, making it easier for the aggregation algorithm to map out the boundaries of safe vs. unsafe zones.
- Feedback Biases: Different feedback types have distinct limitations. Corrections are highly informative about the agent's specific policy but are expensive to collect. Demonstrations are easier to obtain but often fail to cover the exact areas where the agent is likely to struggle.
- Label Propagation: To boost performance with corrections, the authors proposed estimating a transition model of the world to propagate "unacceptable" signals to future states the agent would have visited if the oracle hadn't intervened.
Results
- Predictive Power: The Dawid-Skene aggregation method significantly outperformed standard baselines like Majority Vote, particularly in handling the "action mismatch" noise inherent in demonstrations.
- Safety vs. Efficiency: In the Flappy Bird domain, the "Oracle-in-the-loop" agent achieved rewards comparable to an agent that queries the human 100% of the time, while only querying the human 14-23% of the time.
- AUC Improvements: A classifier trained on data from a suboptimal oracle achieved an AUC of 0.96, compared to just 0.60 for one trained on an optimal oracle, proving that "perfect" data is often less useful in safety contexts.
Conclusion and Future Work
While promising, the authors note challenges in scaling to high-dimensional state spaces like raw pixel data. Additionally, real human oracles introduce complex noise (reflexes, cognitive biases) not fully captured by simulated oracles. Future work suggests using Active Learning to proactively query labels in regions of high uncertainty, rather than relying on passive feedback.
Summary updated