An Optimal Policy Model for Concurrent Uncertainty Estimation During Decision Making

Xiaodong Li, Ruixin Su, Yilin Chen, Tianming YangbioRxiv 2021 · DOI

TL;DR

This paper introduces a novel reaction-time task and an optimal policy model to investigate how we estimate uncertainty during decision-making, not just after. They found that uncertainty estimation is a parallel process, and their model successfully captures individual differences in decision-making behavior by linking perceptual and value-based decisions to a shared evidence accumulation process.

What Problem Does This Paper Address?

We often delay or avoid decisions when faced with uncertainty. While we intuitively understand that uncertainty plays a role in our choices, how we estimate this uncertainty concurrently with making a decision has been difficult to study. Previous research often measured confidence after a decision was made, or used fixed stimulus durations that didn't allow for a dynamic interplay between uncertainty estimation and the decision process itself. This paper aims to bridge this gap by developing a paradigm that disentangles these processes and provides a computational framework to understand their relationship.

Methods

The authors adapted a classic random-dot motion direction discrimination task. In their "optional assistance task," participants could choose to get help from a high-coherence stimulus at any point during their decision-making process. This allowed them to measure both the reaction time for the initial perceptual decision (PD) and the reaction time for the "uncertainty response" (UR), which signifies a choice to seek assistance. They then developed an Optimal Policy Decision Model (OPDM) based on a drift-diffusion process. This model simulates how an agent accumulates evidence and makes decisions by evaluating the expected returns of different actions (wait for more evidence, choose left, choose right, or seek assistance) to maximize their overall reward.

Datasets

The study used behavioral data collected from 10 human participants. No new datasets were introduced.

Key Insights and Ideas

  • Uncertainty is Parallel, Not Post-hoc: The study provides strong evidence that uncertainty estimation happens alongside the perceptual decision-making process, not as an afterthought. The reaction times for seeking assistance (UR) were not dependent on the stimulus's difficulty (coherence), unlike the perceptual decisions themselves.
  • Individual Differences Explained: The optimal policy model, by adjusting parameters like the cost of seeking assistance, successfully captured the significant individual variations observed in participants' reaction times and their propensity to seek help.
  • Shared Neural Substrate: The model suggests that both perceptual decisions and uncertainty estimation might arise from the same underlying evidence accumulation mechanism, hinting at shared neural circuitry.
  • Value-Based Decisions: The model frames uncertainty estimation as a value-based decision, where the agent compares the potential value of seeking assistance against the value of making a direct decision.

Key Results

  • Participants were more likely to seek assistance (UR) when the motion coherence was lower (i.e., when they were more uncertain).
  • The reaction times for seeking assistance (UR) showed significant individual variability and were not strongly correlated with motion coherence, unlike the perceptual decision reaction times.
  • The optimal policy model accurately replicated key behavioral patterns, including choice probabilities, reaction times, and the proportion of uncertainty responses.
  • The model demonstrated that individual differences in behavior could be explained by variations in model parameters, particularly the cost associated with seeking assistance.

Limitations and Future Work

The authors acknowledge that their model predicts a weak correlation between UR reaction time and motion coherence, which was not strongly detected in their experimental data due to noise and a limited number of trials. They suggest future experiments with larger datasets could further test this. They also propose an alternative scenario where URs might be based on estimating motion coherence against a reference point. Future work will focus on exploring the neural mechanisms underlying this concurrent uncertainty estimation, building on the theoretical framework provided by their model.

Summary updated

Abstract

Abstract We often postpone or even avoid making decisions when we feel uncertain. Uncertainty estimation is not an afterthought of decision making but a dynamic process that accompanies decision making in parallel and affects decision making. To study concurrent uncertainty estimation during decision making, we adapted the classic random-dots motion direction discrimination task to allow a reaction-time measure of uncertainty responses. Subjects were asked to judge whether a patch of random dots was moving left or right. In addition, they could seek assistance by choosing to look at a second stimulus that had the same direction but high coherence any time during the task. The task allows us to measure the reaction time of both the perceptual decisions and the uncertainty responses. The subjects were more likely to choose the uncertainty response when the motion coherence was low, while their reaction times of the uncertainty responses showed individual variations. To account for the subjects’ behavior, we created an optimal policy decision model in which decisions are based on the value functions computed from the accumulated evidence using a drift-diffusion process. Model simulations captured key features of the subjects’ choices, reaction times, and proportions of uncertainty responses. Varying model parameters explained individual variations in the subjects and the correlations between decision accuracy, proportions of uncertainty responses, and reaction times at the individual level. Our model links perceptual decisions and value-based decisions and indicates that concurrent uncertainty estimation may be based on comparisons between values of uncertainty responses and perceptual decisions, both of which may be derived from the same evidence accumulation process during decision making. It provides a theoretical framework for future investigations, including the ones that aim at the underlying neural mechanism.
BibTeX
@misc{li2021optimal,
  author = {Xiaodong Li and Ruixin Su and Yilin Chen and Tianming Yang},
  title = {{An Optimal Policy Model for Concurrent Uncertainty Estimation During Decision Making}},
  year = {2021},
  doi = {10.1101/2021.10.14.464349},
  howpublished = {bioRxiv},
}

Loading related content...

Figure