Abstract
This paper considers agency in the setting of embodied or active inference. In brief, we associate a sense of agency with prior beliefs about action and ask what sorts of beliefs underlie optimal behavior. In particular, we consider prior beliefs that action minimizes the Kullback-Leibler (KL) divergence between desired states and attainable states in the future. This allows one to formulate bounded rationality as approximate Bayesian inference that optimizes a free energy bound on model evidence. We show that constructs like expected utility, exploration bonuses, softmax choice rules and optimism bias emerge as natural consequences of this formulation. Previous accounts of active inference have focused on predictive coding and Bayesian filtering schemes for minimizing free energy. Here, we consider variational Bayes as an alternative scheme that provides formal constraints on the computational anatomy of inference and action—constraints that are remarkably consistent with neuroanatomy. Furthermore, this scheme contextualizes optimal decision theory and economic (utilitarian) formulations as pure inference problems. For example, expected utility theory emerges as a special case of free energy minimization, where the sensitivity or inverse temperature (of softmax functions and quantal response equilibria) has a unique and Bayes-optimal solution—that minimizes free energy. This sensitivity corresponds to the precision of beliefs about behavior, such that attainable goals are afforded a higher precision or confidence. In turn, this means that optimal behavior entails a representation of confidence about outcomes that are under an agent's control.
Introduction: Choice as Inference
In traditional economics and psychology, perception and decision-making are often treated as distinct systems: one system infers the state of the world, while another decides how to act based on a separate utility function. This paper argues that this separation is unnecessary. Instead, it proposes that "agency" and choice are part of a unified "pure inference problem." By minimizing the difference between desired future states and attainable ones, the brain naturally generates complex behaviors—such as exploration, economic utility, and even "optimism bias"—without requiring separate, ad-hoc modules for each.
The Active Inference Framework
The core of this proposal is Active Inference, a framework where biological systems navigate the world by minimizing Variational Free Energy (a proxy for surprise or prediction error). The agent maintains a generative model of the world and acts to ensure its observations match its prior beliefs about the states it should occupy.
In this framework, the agent's "drive" is to minimize the Kullback–Leibler (KL) divergence between its goals (expected states) and its reach (attainable states). This mathematical necessity naturally decomposes into two terms: Expected Utility (exploitation of known rewards) and Entropy (exploration of the unknown). Consequently, curiosity and exploration are not separate drives but inherent requirements for minimizing long-term surprise.
Methods: Variational Bayes and MDPs
To make this framework biologically plausible, the authors move away from continuous-time predictive coding toward Variational Bayes applied to discrete-time Markov Decision Processes (MDPs). They employ a "mean-field approximation," assuming the brain factorizes complex probability distributions into simpler, independent components (e.g., separating beliefs about the "state" from beliefs about "precision").
This approach derives a set of message-passing equations that resemble neural network updates. "Neurons" in this model pass expectations back and forth until they converge on a consistent internal model of the world and the agent's place within it.
Simulating Choice: The Limited Offer Game
To validate the model, the authors simulated a "Limited Offer" game. In this scenario, an agent must decide how long to wait for a "high offer" while facing the risk that a "low offer" currently available might be withdrawn at any moment.
The simulation successfully demonstrated emergent economic behaviors. The agent showed "patient" behavior (waiting for high offers) and "temporal discounting" (valuing immediate offers more as the risk of withdrawal increased). Crucially, these behaviors were not programmed in via utility functions but emerged purely from the math of inference and prior beliefs about goals.
Key Insights and Neuroanatomy
One of the most striking insights is the reinterpretation of Dopamine. Rather than a simple reward signal, the authors suggest dopamine encodes the precision (confidence) of our beliefs about our policies. When a clear path to a goal is identified, dopamine spikes because the agent's confidence in its chosen action has increased.
Furthermore, the authors map these Variational Bayes updates onto specific brain regions:
- Prefrontal Cortex: Responsible for updating beliefs about the current state (Perception).
- Striatum: Responsible for updating beliefs about which policy to follow (Action Selection).
- VTA/Substantia Nigra (Dopamine): Responsible for updating the precision or confidence of those policies (Evaluation).
Conclusion
By treating agency as a consequence of prior beliefs, this framework provides a unified account of bounded rationality. It suggests that our sense of control is a probabilistic belief, and that "value" is simply a prior expectation of the states we believe we should occupy. This perspective offers a powerful new lens for understanding both healthy decision-making and psychiatric conditions where the balance of precision and inference may be disrupted.
Summary updated