TL;DR
This paper introduces a recurrent neural network (RNN) model that learns to make flexible and adaptive decisions by predicting future events based on sequences of sensory, action, and reward information. The model successfully replicates animal and human behavior across various decision-making tasks, offering a unified framework for understanding brain computations underlying flexible behavior.
What Problem Does This Paper Address?
The brain's remarkable ability to navigate complex and dynamic environments hinges on its capacity for flexible and adaptive decision-making. This involves not only processing current sensory inputs to choose appropriate actions but also anticipating the consequences of those actions and learning from outcomes. Existing computational models often focus on specific aspects of decision-making, such as evidence accumulation or reinforcement learning, but struggle to integrate these components into a unified framework that captures the full spectrum of adaptive behavior. This paper aims to bridge this gap by proposing a single RNN model that learns complex contingencies between sensory, action, and reward events to achieve flexible and adaptive decision-making.
Methods
The core of the proposed model is a Gated Recurrent Unit (GRU) network, a type of RNN known for its effectiveness in learning sequential data. The GRU network is chosen for its ability to handle long-term dependencies in sequences, crucial for learning complex contingencies. The model is trained using supervised learning to predict future events (sensory, action, and reward) based on past event sequences. A key innovation is the inclusion of sensory, action, and reward events in the input sequences, allowing the network to learn the interdependencies between them. The model's output units mirror the input units, predicting future events. The training objective is to minimize the difference between the predicted and actual future event sequences.
Datasets
The authors evaluate their model on four exemplary tasks previously used in animal and human experiments:
- Probabilistic Reasoning Task: Used to study sequential decision-making and compare model behavior and neural activity with experimental findings.
- Multi-sensory Integration Task: Adapted from Gu et al. (2008) to demonstrate Bayesian inference capabilities.
- Post-decision Wagering Task: Used to investigate meta-cognition, specifically confidence.
- Two-step Task: An adapted version of the task from Daw et al. (2011) to assess adaptive behavior in changing environments.
The training datasets for these tasks were generated using simulated trials, often incorporating elements like drift-diffusion models with collapsing bounds to mimic experimental paradigms.
Key Insights and Ideas
- Unified Framework: The model demonstrates that a single GRU network, trained on sequences of sensory, action, and reward events, can account for a wide range of decision-making phenomena previously addressed by separate models.
- Beyond Reinforcement Learning: The model achieves adaptive behavior through sequence learning and prediction, offering an alternative to traditional reinforcement learning approaches that heavily rely on reward prediction errors.
- Neural Plausibility: The internal dynamics of the network, particularly the "ramping" activity patterns observed in some units, closely mimic findings from neurophysiological studies of decision-making neurons.
- Bayesian Inference: The model naturally performs Bayesian inference without explicit programming, demonstrating how sequential learning can lead to probabilistic reasoning.
- Meta-cognition: The model shows potential for supporting meta-cognitive functions like confidence, as evidenced by its performance on the post-decision wagering task.
Key Results
- Probabilistic Reasoning: The model reproduced animal choice and reaction time patterns and exhibited unit activities similar to parietal neurons involved in evidence accumulation.
- Multi-sensory Integration: The model generalized to bi-modal inputs, achieving higher accuracy and demonstrating Bayesian inference capabilities, matching experimental predictions.
- Confidence: The model's behavior on the post-decision wagering task was consistent with monkeys' strategies, suggesting it can model confidence.
- Adaptive Behavior: In the two-step task, the model adapted to reversed reward contingencies, showing a behavior closer to model-based decision-making.
- Speed-Accuracy Tradeoff: The model identified distinct unit populations ("when" and "which" units) contributing to reaction time and choice, offering a potential mechanism for modulating the speed-accuracy tradeoff.
Limitations and Future Work
- Model Complexity: While the GRU network is powerful, the authors acknowledge that the current model uses a relatively small number of units (128) and has only been tested on "small-scale toy problems."
- Loss Function: The authors suggest that the particular loss function used, which treats all events equally, could be improved by normalizing within domains or weighting events differently based on their temporal distance from the reward.
- Neural Circuitry: The paper speculates on which brain structures might implement these computations (e.g., prefrontal cortex, basal ganglia) and suggests future work could incorporate more detailed basal ganglia circuitry.
- Natural Language Processing: The authors note the parallelism between their model and approaches used in natural language processing, suggesting that expanding the network with more units and layers could enable modeling of more complex behaviors, including natural language processing.
- Dopamine System: The role of the dopamine system in signaling reward prediction errors and potentially other events is discussed, with future experiments suggested to test its fit with the model's predictions.
Summary updated