Abstract
Diagnosing the source of failure in autonomous agents and human experts is critical for effective intervention. This paper presents the Generative Error Model (GEM), a probabilistic framework that disambiguates between representational "blind spots"—features of the environment the actor cannot perceive—and execution noise. By treating the actor's internal state as a latent variable, GEM performs Bayesian inference to identify the most likely cause of observed errors. Evaluations in a simulated gridworld and a human kitchen task show that GEM accurately recovers representational flaws and that explicitly modeling execution noise is essential to prevent the misidentification of random errors as systematic blind spots.
Introduction
When an AI system or a human expert fails at a task, the root cause is often opaque. For instance, if a self-driving car misses a turn, did it fail to perceive the street sign due to a sensor limitation (a representational error), or did it see the sign but fail to execute the correct maneuver (an execution error)?
Existing methods in Reinforcement Learning (RL) and Cognitive Science typically focus on predicting future actions or inferring intent, but they rarely diagnose the underlying source of failure. Disambiguating these error types is critical because the remedy for a representational error—such as adding a new sensor—is fundamentally different from the remedy for an execution error, which might require additional training or attention support.
The Generative Error Model (GEM)
The authors introduce the Generative Error Model (GEM), a probabilistic graphical model that treats an actor's internal state as a latent variable. GEM models the decision-making process by incorporating several key components:
- Blind Spots ($b$): A binary mask over the true features of the environment. If a feature is designated as a blind spot, the actor is unable to perceive it.
- Implicit State ($s$): Because the actor cannot perceive all features, they must make assumptions about missing information. GEM models this "implicit view" of the world.
- Execution Noise ($\eta$): A parameter representing the frequency with which an actor deviates from an optimal policy, even when their representation is correct.
To identify the most likely causes of error, the framework employs Bayesian inference. For simple environments, exact variable elimination is used. In more complex scenarios with large state spaces, Collapsed Gibbs Sampling is utilized to approximate the posterior distribution of blind spots and noise.
Experimental Evaluation
The GEM framework was evaluated across two distinct domains:
- Simulated Gridworld: A 10x10 environment where an RL agent collects green objects while avoiding red ones. A "color-blind" agent was simulated to test if GEM could correctly identify the "color" feature as the blind spot.
- Kitchen Task (Human Study): A mock kitchen environment where participants prepared dishes from memory. To create a controlled blind spot, salt and sugar were made visually indistinguishable. A dataset of 2,420 state-action-error tuples was collected to evaluate the model's diagnostic capabilities.
Results and Key Insights
GEM demonstrated strong performance in identifying representational errors. In the Gridworld simulation, the model identified the color blind spot with 70% probability after observing only 100 data points. In the more complex Kitchen Task, GEM achieved 30% accuracy in identifying the blind spot vector—a significant feat given that the probability of guessing the vector by chance is only 0.002.
Noise as a "Safety Valve"
A pivotal finding of this research is the necessity of modeling execution noise. Even if the exact noise level cannot be perfectly estimated, its presence in the model acts as a "safety valve." Without a noise parameter, the model tends to interpret every stochastic or random mistake as a systematic representational failure, leading to "hallucinated" blind spots.
Recovering the Implicit View
Beyond identifying missing features, GEM can infer the actor's internal assumptions. In the Gridworld task, the model correctly inferred that the agent acted as if every perceived object was green. In the Kitchen Task, it successfully deduced that when humans made errors with salt, they were likely assuming the container held sugar, reflecting the physical setup of the experiment.
Conclusion and Future Work
GEM provides a versatile tool for diagnosing flaws in both RL agents and human actors, offering significant potential for safety auditing and human-AI teaming. However, several limitations remain. The current model requires an "oracle" observer with knowledge of the true state and optimal policy, which may not be available in all real-world settings. Additionally, the model currently treats blind spots as static and binary. Future work could extend GEM to handle continuous visibility (e.g., lighting conditions) and state-dependent blind spots (e.g., viewing angles), as well as improving computational efficiency for high-dimensional data like raw video.
Summary updated