Introduction
Understanding the internal motivations and world models of biological agents—such as animals or humans—is a fundamental challenge in neuroscience and artificial intelligence. Traditional Inverse Reinforcement Learning (IRL) often assumes that an agent is an expert performing optimally within a fully observable environment. However, real-world behavior is frequently characterized by "mistakes" that arise not from irrationality, but from acting rationally based on imperfect information or flawed internal assumptions about physics and rewards.
This paper introduces Inverse Rational Control (IRC), a novel framework designed to "reverse-engineer" an agent's internal model from its observed actions. By shifting the focus from absolute optimality to subjective rationality, IRC can recover hidden parameters and belief dynamics even in complex, continuous, and partially observable environments.
The IRC Framework
The IRC approach addresses two primary hurdles: partial observability and continuous complexity. Biological agents act based on beliefs—internal estimates of the world state—which are hidden from observers. Furthermore, real-world control involves nonlinear dynamics that make standard discrete IRL methods computationally prohibitive.
IRC employs a two-loop architecture to solve this:
- The Forward Loop (Policy Ensembles): Rather than solving for a single optimal policy, the framework learns a generalized policy over a manifold of possible models. Using Deep Deterministic Policy Gradient (DDPG), an actor-critic reinforcement learning algorithm, the authors train a neural network to output the optimal action for any given set of model parameters, such as movement gain or noise levels.
- The Inverse Loop (Maximum Likelihood): Given a sequence of observed trajectories, the framework searches the pre-learned policy manifold to identify the specific parameters ($\theta$) that maximize the likelihood of the observed behavior.
To account for the agent's internal state estimation, IRC utilizes an Extended Kalman Filter (EKF). The observer performs a "hard E-step" (Maximum A Posteriori inference) to estimate the agent's likely internal beliefs based on its actual movements.
Experimental Paradigm: Catching Fireflies
The authors evaluated IRC using the "Catching Fireflies" task, a naturalistic paradigm common in neuroscience. In this task, an agent must navigate to a target (a "firefly") that disappears shortly after the trial begins. The agent must rely on noisy internal estimates of its own motion (optic flow) to reach the target's remembered location.
The task was implemented in two versions: a simplified 1D Firefly environment with linear dynamics, and a complex 2D Firefly environment featuring nonlinear dynamics (angular and forward velocity) and continuous control.
Key Results
The IRC framework demonstrated high precision in recovering the internal parameters of simulated agents where ground truth was known.
Parameter Recovery
In the 2D task, the estimated parameters—including forward velocity gain, angular velocity gain, and process noise—closely matched the true parameters. The estimates typically fell within the 95% confidence intervals derived from Fisher information, indicating that the framework can accurately distinguish between different internal models based solely on behavioral data.
Belief Dynamics
Beyond static parameters, the framework successfully reconstructed the agent's hidden "belief" trajectories. The estimated beliefs regarding position and velocity were nearly identical to the agent's actual internal states. This capability allows researchers to interpret behavioral variability as a consequence of specific internal biases rather than mere stochastic noise.
Discussion and Future Work
IRC provides a mathematical implementation of "Theory of Mind," allowing researchers to model a subject's internal mental state. However, the framework currently assumes that the agent's internal model belongs to a predefined parametric family. If an agent employs a fundamentally different strategy, the results may be biased.
Future research could extend IRC to hierarchical models where internal parameters might evolve over time, enabling the study of learning and adaptation. Additionally, more sophisticated filtering techniques could be integrated to handle highly complex or multi-modal belief distributions.
Summary updated