Blind Spot Detection for Safe Sim-to-Real Transfer

Ramya Ramakrishnan, Ece Kamar, Debadeepta Dey, Eric Horvitz, Julie ShahJAIR 2020 · DOI · OpenReview

Abstract

Reinforcement learning agents often fail during "sim-to-real" transfer because they encounter "blind spots"—regions where their simplified internal representation cannot distinguish between safe and dangerous real-world states. This paper introduces a framework that uses limited human feedback to train a predictive model of these blind spots, allowing agents to proactively "ask for help" before making a catastrophic mistake.

Introduction: The Sim-to-Real Gap

The "sim-to-real" gap is a notorious hurdle in robotics and autonomous systems. While training in simulation is cost-effective and safe, simulators are inevitably simpler than reality. This research focuses on a specific type of failure: representational incompleteness.

Conceptual overview of agent blind spots where representational incompleteness leads to unacceptable actions.

Imagine an autonomous car trained in a simulator that has never seen an ambulance. In the real world, the agent might perceive an ambulance simply as a "large white vehicle." Because its internal model lacks features like "siren" or "emergency lights," it treats the ambulance exactly like a standard delivery truck. If the agent continues driving normally, it makes a high-cost error. The agent isn't just wrong; it’s confident and wrong because it literally cannot see the features that necessitate a different action.

Methodology

The authors formalize blind spot detection as a noisy supervised learning problem with class imbalance. The proposed pipeline consists of three main stages:

  1. Data Collection via Oracles: The agent interacts with an "oracle" (a human or expert system) in the real world. Feedback is gathered through Corrections (the oracle monitors the agent and intervenes) or Demonstrations (the oracle shows the agent how to perform the task).
  2. Label Aggregation (Dawid-Skene): Due to the representational gap, one agent observation might correspond to multiple real-world states—some safe, some not. This creates "noise" in the labels. The authors use the Dawid-Skene algorithm (an Expectation-Maximization approach) to estimate label reliability and aggregate them into a "ground truth" probability that a specific observation is a blind spot.
  3. Model Learning and Calibration: A Random Forest classifier is trained to generalize from visited states to unseen regions of the state space. Since blind spots are rare, the researchers employ oversampling and temperature calibration to ensure the model’s probability estimates are useful for decision-making.

Finally, the system implements Oracle-in-the-loop (OIL) execution: during deployment, if the model predicts a high probability of a blind spot, the agent pauses and queries the oracle for the correct action.

Experimental Evaluation

The approach was evaluated on two modified classic environments designed to simulate representational gaps:

  • Modified Catcher: The agent must catch falling fruit but lacks the features to distinguish between "good" fruit and "bad" (dangerous) fruit that appears identical in its representation.
  • Modified Flappy Bird: The agent must fly between pipes. Some pipes are made of copper (creating dangerous wind conditions if the bird flies high), but the agent cannot "see" the pipe material.

Key Insights

  • The Suboptimality Paradox: A surprising finding is that suboptimal oracles and simulator policies actually improve the model. If an oracle is "too perfect," it avoids dangerous regions entirely, leaving the agent with no data about its blind spots. Suboptimal behavior provides a more diverse set of labels, making it easier for the aggregation algorithm to map out the boundaries of safe vs. unsafe zones.
  • Feedback Biases: Different feedback types have distinct limitations. Corrections are highly informative about the agent's specific policy but are expensive to collect. Demonstrations are easier to obtain but often fail to cover the exact areas where the agent is likely to struggle.
  • Label Propagation: To boost performance with corrections, the authors proposed estimating a transition model of the world to propagate "unacceptable" signals to future states the agent would have visited if the oracle hadn't intervened.

Results

  • Predictive Power: The Dawid-Skene aggregation method significantly outperformed standard baselines like Majority Vote, particularly in handling the "action mismatch" noise inherent in demonstrations.
  • Safety vs. Efficiency: In the Flappy Bird domain, the "Oracle-in-the-loop" agent achieved rewards comparable to an agent that queries the human 100% of the time, while only querying the human 14-23% of the time.
  • AUC Improvements: A classifier trained on data from a suboptimal oracle achieved an AUC of 0.96, compared to just 0.60 for one trained on an optimal oracle, proving that "perfect" data is often less useful in safety contexts.

Conclusion and Future Work

While promising, the authors note challenges in scaling to high-dimensional state spaces like raw pixel data. Additionally, real human oracles introduce complex noise (reflexes, cognitive biases) not fully captured by simulated oracles. Future work suggests using Active Learning to proactively query labels in regions of high uncertainty, rather than relying on passive feedback.

Summary updated

Abstract

Agents trained in simulation may make errors when performing actions in the real world due to mismatches between training and execution environments. These mistakes can be dangerous and difficult for the agent to discover because the agent is unable to predict them a priori. In this work, we propose the use of oracle feedback to learn a predictive model of these blind spots in order to reduce costly errors in real-world applications. We focus on blind spots in reinforcement learning (RL) that occur due to incomplete state representation: when the agent lacks necessary features to represent the true state of the world, and thus cannot distinguish between numerous states. We formalize the problem of discovering blind spots in RL as a noisy supervised learning problem with class imbalance. Our system learns models for predicting blind spots within unseen regions of the state space by combining techniques for label aggregation, calibration, and supervised learning. These models take into consideration noise emerging from different forms of oracle feedback, including demonstrations and corrections. We evaluate our approach across two domains and demonstrate that it achieves higher predictive performance than baseline methods, and also that the learned model can be used to selectively query an oracle at execution time to prevent errors. We also empirically analyze the biases of various feedback types and how these biases influence the discovery of blind spots. Further, we include analyses of our approach that incorporate relaxed initial optimality assumptions. (Interestingly, relaxing the assumptions of an optimal oracle and an optimal simulator policy helped our models to perform better.) We also propose extensions to our method that are intended to improve performance when using corrections and demonstrations data.
BibTeX
@article{ramakrishnan2020blind,
  author = {Ramya Ramakrishnan and Ece Kamar and Debadeepta Dey and Eric Horvitz and Julie Shah},
  title = {{Blind Spot Detection for Safe Sim-to-Real Transfer}},
  journal = {J. Artif. Intell. Res.},
  year = {2020},
  doi = {10.1613/jair.1.11436},
}
Figure