Reward Optimization in the Primate Brain: A Probabilistic Model of Decision Making under Uncertainty

Yanping Huang, Rajesh P. N. RaoPLoS 2013 · DOI

Abstract

A key problem in neuroscience is understanding how the brain makes decisions under uncertainty. Important insights have been gained using tasks such as the random dots motion discrimination task in which the subject makes decisions based on noisy stimuli. A descriptive model known as the drift diffusion model has previously been used to explain psychometric and reaction time data from such tasks but to fully explain the data, one is forced to make ad-hoc assumptions such as a time-dependent collapsing decision boundary. We show that such assumptions are unnecessary when decision making is viewed within the framework of partially observable Markov decision processes (POMDPs). We propose an alternative model for decision making based on POMDPs. We show that the motion discrimination task reduces to the problems of (1) computing beliefs (posterior distributions) over the unknown direction and motion strength from noisy observations in a bayesian manner, and (2) selecting actions based on these beliefs to maximize the expected sum of future rewards. The resulting optimal policy (belief-to-action mapping) is shown to be equivalent to a collapsing decision threshold that governs the switch from evidence accumulation to a discrimination decision. We show that the model accounts for both accuracy and reaction time as a function of stimulus strength as well as different speed-accuracy conditions in the random dots task.

Introduction

For decades, the Random Dots Motion (RDM) task has served as a primary paradigm for studying how the brain accumulates evidence. In this task, subjects must determine the net direction of moving dots amidst noise. While the standard Drift Diffusion Model (DDM) effectively describes evidence "drifting" toward a threshold, it requires ad-hoc assumptions—like a "collapsing" decision boundary—to explain why subjects eventually commit to a choice even when evidence is ambiguous.

This research moves beyond descriptive models to a normative framework, asking why the brain would adopt such a strategy. The authors propose that these behaviors are the mathematically optimal result of maximizing future rewards under uncertainty.

The POMDP Framework

The authors model decision-making as a Partially Observable Markov Decision Process (POMDP). In this setup, the "true" state of the stimulus (direction and strength) is hidden from the agent. The agent (the brain) must rely on noisy sensory observations—modeled as spikes from MT neurons—to update its internal representation of the world.

The POMDP framework for decision making. The agent uses Bayesian inference to update its belief state ($b_t$) based on noisy observations and rewards, then selects an action ($a_t$) according to an optimal policy $\pi$ to maximize future rewards.

At each time step, the agent performs two tasks:

  1. Bayesian Inference: Updating a posterior distribution (belief state) over the possible motion directions.
  2. Action Selection: Choosing between three actions: move eyes Left ($A_L$), move eyes Right ($A_R$), or continue sampling data ($A_S$).

The "optimal policy" is the mapping of beliefs to actions that maximizes the expected sum of future rewards, accounting for correct/incorrect outcomes and the "sampling cost" of time.

Results: The Emergence of Collapsing Boundaries

The model demonstrates that a collapsing decision threshold is not a biological quirk but a direct consequence of reward maximization. Early in a trial, the potential gain in accuracy from waiting is high. However, as time passes, the cumulative cost of sampling grows, and the marginal benefit of additional data diminishes. Eventually, it becomes more "profitable" to make a choice than to continue waiting.

Optimal policy and decision boundaries. (a) Value function over time. (b) Policy regions showing the transition from sampling (red) to decision (blue/green). (c) The resulting collapsing boundaries for various reward-to-cost ratios.

This optimal policy is governed by a single "Reward Ratio": $(RN - RP) / RS$, where $RN$ and $RP$ are rewards for correct/incorrect choices and $RS$ is the sampling cost. By adjusting this ratio, the model successfully fits human and monkey data across different speed-accuracy conditions. Notably, the model's predicted neural buildup rate of 227.7 spikes/s² per unit of coherence closely matches the 222.5 spikes/s² observed in actual LIP neuron recordings.

Conclusion

By framing decision-making as a POMDP, the authors provide a first-principles explanation for the "collapsing boundaries" observed in behavioral data. This normative approach suggests that the primate brain is an efficient optimizer, naturally balancing the need for accuracy against the inherent cost of time. Future work will look into how the brain learns these optimal policies, likely through the dopamine-mediated temporal difference learning system.

Summary updated

Abstract

A key problem in neuroscience is understanding how the brain makes decisions under uncertainty. Important insights have been gained using tasks such as the random dots motion discrimination task in which the subject makes decisions based on noisy stimuli. A descriptive model known as the drift diffusion model has previously been used to explain psychometric and reaction time data from such tasks but to fully explain the data, one is forced to make ad-hoc assumptions such as a time-dependent collapsing decision boundary. We show that such assumptions are unnecessary when decision making is viewed within the framework of partially observable Markov decision processes (POMDPs). We propose an alternative model for decision making based on POMDPs. We show that the motion discrimination task reduces to the problems of (1) computing beliefs (posterior distributions) over the unknown direction and motion strength from noisy observations in a bayesian manner, and (2) selecting actions based on these beliefs to maximize the expected sum of future rewards. The resulting optimal policy (belief-to-action mapping) is shown to be equivalent to a collapsing decision threshold that governs the switch from evidence accumulation to a discrimination decision. We show that the model accounts for both accuracy and reaction time as a function of stimulus strength as well as different speed-accuracy conditions in the random dots task.
BibTeX
@inproceedings{huang2013reward,
  author = {Yanping Huang and Rajesh P. N. Rao},
  title = {{Reward Optimization in the Primate Brain: A Probabilistic Model of Decision Making under Uncertainty}},
  booktitle = {PLoS},
  year = {2013},
  doi = {10.1371/journal.pone.0053344},
}
Figure