Abstract
In many real-world domains, there is a gap between what an agent intends to do and what they actually do. This paper introduces a Bayesian framework to estimate an agent's execution skill—their ability to carry out a planned action precisely—by observing their behavior in continuous environments. Unlike previous methods that relied on rewards, this approach analyzes the actions themselves, allowing for faster convergence and the ability to model agents that aren't perfectly rational.
Introduction: The "Shaky Hand" Problem
In domains ranging from robotics to professional sports, an agent's performance is a combination of two distinct factors: Strategic Skill (the ability to pick the best intended action) and Execution Skill (the physical precision to carry out that intention). This discrepancy is often referred to as "execution noise."
For an outside observer, the challenge lies in the fact that we only see the final executed action ($x$), the state ($s$), and the resulting reward ($r$). The agent's internal intended action ($t$) remains hidden. If a billiards player misses a shot, was it a failure of strategy (aiming at the wrong spot) or a failure of execution (a "shaky hand")? This research focuses on isolating and estimating that execution skill ($\sigma$) from observation alone.
Methodology: The Bayesian Approach (TBA)
The authors propose the Bayesian Approach (TBA), modeling the interaction using a Bayesian Network with three primary nodes:
- $\Sigma$ (Sigma): The agent's execution skill level, represented as the standard deviation of their execution noise.
- $T$ (Target): The action the agent intended to take.
- $X$ (Executed): The action actually performed and observed.
The core of the method involves calculating the posterior probability of a skill level given an observed action: $P(\sigma | x) \propto P(x | \sigma)P(\sigma)$. Since the intended action $T$ is latent, the model marginalizes over it: $$P(x | \sigma) = \int_{t \in T} P(t | \sigma)P(x | t, \sigma) dt$$
Beyond Perfect Rationality
A key innovation of this work is moving beyond the assumption of "perfect rationality." Previous methods assumed agents always intended the mathematically optimal action. TBA introduces a "focal action" variant, assuming the agent selects from a set of "sensible" actions (e.g., any shot in billiards that might pocket a ball). This makes the model robust to agents that make strategic errors.
Experimental Evaluation
The method was evaluated in two simulated environments:
- 1D-Darts: A controlled 1D continuous action space where rewards alternate between 0 and 1, used to compare against traditional reward-based methods.
- Computational Billiards (8-ball): A complex domain using the FastFiz physics simulator. Tests involved both high-level AI (CueCard) and random agents to evaluate how the model handles varying levels of strategic skill.
Key Results and Insights
The Bayesian approach demonstrates significant advantages over traditional "Observed Reward" (OR) methods:
- Information Density: By analyzing actions rather than just rewards, TBA extracts more information per observation. In the 1D-Darts domain, TBA converged to the correct skill level two orders of magnitude faster than reward-based methods.
- Accuracy and Robustness: In billiards, TBA performance closely tracked the "True Noise" baseline—the theoretical limit where the observer knows the intended action. It successfully ranked agents by execution skill even when those agents had poor strategic planning.
- Rapid Differentiability: The model can distinguish between different skill levels in as few as 50 observations. In a game like 8-ball, this allows for a solid estimate of an opponent's physical skill after just a few games.
Conclusion and Future Work
This research provides a robust framework for disentangling execution noise from strategic intent. While currently limited by assumptions of independent, zero-mean noise and the need for some prior knowledge of agent strategy, it paves the way for more sophisticated models. Future work aims to simultaneously estimate both strategic and execution skill from scratch and apply these techniques to real-world data, such as video analysis of human athletes.
Summary updated