Optimal multisensory decision-making in a reaction-time task

Jan Drugowitsch, Gregory C. DeAngelis, Eliana M. Klier, Dora E. Angelaki, Alexandre PougeteLife 2014 · DOI

TL;DR

This paper tackles how humans make decisions when integrating information from multiple senses, specifically when they can choose when to respond. They found that traditional optimality measures fail in these "reaction-time" tasks, showing apparent sub-optimality. However, by developing a novel diffusion model that accounts for both speed and accuracy, they demonstrate that humans do integrate sensory evidence near-optimally over time and across modalities, even when controlling their own decision speed.

What Problem Does This Paper Address?

The field of multisensory integration has largely studied how combining information from different senses (like vision and vestibular input) improves our ability to perceive and make decisions. Previous research, however, has primarily used tasks where the stimulus duration is fixed, and participants are forced to wait until the stimulus disappears before responding. This setup simplifies analysis but doesn't reflect real-world scenarios where we often decide quickly as soon as we have enough information. The key problem this paper addresses is whether the "near-optimal" integration observed in fixed-duration tasks holds when subjects can control their own decision time (reaction-time tasks), and how sensory inputs with varying reliability over time affect this. Traditional optimality metrics, which focus solely on accuracy, fail to capture this complexity, leading to apparent sub-optimality in reaction-time tasks.

Methods

The authors developed a novel extended diffusion model (DM). Standard DMs are effective for single-sensory decision-making, modeling evidence accumulation over time until a decision bound is reached. This new model extends DMs in two crucial ways:

  1. Multisensory Integration: It optimally combines evidence from multiple sensory modalities (visual and vestibular) by weighting each cue's contribution according to its reliability. This is achieved by combining the sensitivities of individual cues using a specific mathematical relationship (Equation 2 in the paper).
  2. Time-Varying Reliability: It accounts for sensory inputs whose reliability changes over time. This is modeled by weighting the momentary evidence from each cue by its time-varying sensitivity profile (e.g., visual cue sensitivity is weighted by velocity, vestibular by acceleration).

They then fitted this model to behavioral data from human subjects performing a heading discrimination task. The model's parameters (like decision bounds and cue sensitivities) were optimized to best explain both the choices subjects made and their reaction times.

Datasets

The study used behavioral data collected from seven human subjects performing a visual/vestibular heading discrimination task. This task was a reaction-time version of a previously established paradigm. Some subjects also participated in a follow-up experiment with more coherence levels. The paper does not introduce a new benchmark dataset but rather uses human behavioral data to validate their model.

Key Insights and Ideas

  • Reaction Time is Crucial for Optimality: Traditional metrics that ignore reaction time can make behavior appear sub-optimal, even when it's not. The trade-off between speed and accuracy is a critical factor.
  • Optimal Cue Combination is Dynamic: Humans don't just average sensory inputs; they optimally weight them based on their current reliability. This weighting can change over time.
  • Sensory Evidence is Integrated Before Decision: The model's success supports the idea that sensory information from different modalities is combined into a common representation before a decision is made, rather than through independent "parallel race" processes.
  • Suppressing Noisy Information: When visual information is unreliable (low coherence), subjects can effectively suppress it and rely more heavily on the vestibular cue, demonstrating sophisticated cue weighting.

Key Results

  • Apparent Sub-optimality: In a reaction-time task, subjects showed higher discrimination thresholds (worse accuracy) in the combined visual-vestibular condition compared to what traditional optimal cue integration models predicted, especially at higher coherences. This was often because they made decisions faster.
  • Model Explains Behavior: The extended diffusion model, which accounts for both speed and accuracy, quantitatively explained subjects' choices and reaction times across various conditions. The model's predictions for combined cue sensitivity closely matched observed data.
  • Near-Optimal Integration: The model's success strongly supports the hypothesis that subjects integrate sensory evidence near-optimally over time and across modalities, even when controlling their decision speed.
  • Rejection of Parallel Race Models: The data and model analysis decisively ruled out simpler "parallel race" models, which assume independent processing of cues.

Limitations and Future Work

  • Assumed Knowledge of Sensitivities: The model assumes subjects can estimate cue sensitivities, which is reasonable given the task design but might not hold in all naturalistic scenarios.
  • Ideal Observer Assumption: The model represents an "upper bound" on performance, assuming subjects have perfect knowledge of their sensitivities and temporal profiles. Real subjects might have less precise estimates.
  • Future Work: The authors suggest extending this framework to predict neural responses in decision-making areas, which would allow for direct testing of their model's predictions about temporal evidence integration. They also plan to explore how sensitivity estimation and cue integration are implemented neurally.

Summary updated

Abstract

Humans and animals can integrate sensory evidence from various sources to make decisions in a statistically near-optimal manner, provided that the stimulus presentation time is fixed across trials. Little is known about whether optimality is preserved when subjects can choose when to make a decision (reaction-time task), nor when sensory inputs have time-varying reliability. Using a reaction-time version of a visual/vestibular heading discrimination task, we show that behavior is clearly sub-optimal when quantified with traditional optimality metrics that ignore reaction times. We created a computational model that accumulates evidence optimally across both cues and time, and trades off accuracy with decision speed. This model quantitatively explains subjects's choices and reaction times, supporting the hypothesis that subjects do, in fact, accumulate evidence optimally over time and across sensory modalities, even when the reaction time is under the subject's control.
BibTeX
@inproceedings{drugowitsch2014optimal,
  author = {Jan Drugowitsch and Gregory C. DeAngelis and Eliana M. Klier and Dora E. Angelaki and Alexandre Pouget},
  title = {{Optimal multisensory decision-making in a reaction-time task}},
  booktitle = {eLife},
  year = {2014},
  doi = {10.7554/elife.03005},
}
Figure