TL;DR
This paper introduces a novel "two-stream" framework for sequential decision-making agents that models positive and negative reward processing separately. This allows for more flexible and human-like behavior, particularly in capturing the diverse strategies observed in individuals with various neurological and psychiatric disorders, while unifying models across Multi-Armed Bandits (MAB), Contextual Bandits (CB), and Reinforcement Learning (RL).
What Problem Does This Paper Address?
Traditional artificial agents are often optimized for consistent, maximal reward accumulation. However, human decision-making is far more nuanced, exhibiting varied strategies and learning trajectories that can lead to similar outcomes. This variability is particularly pronounced in individuals with neurological and psychiatric disorders, where reward processing mechanisms are known to be disrupted. Existing computational models often struggle to capture this diversity. The authors aim to bridge this gap by developing a more flexible framework that can model these different behavioral patterns and, in doing so, create more robust and interpretable agents. They specifically target the challenge of unifying MAB, CB, and RL under a single, flexible model that can account for individual differences in how rewards and punishments are processed.
Methods
The core innovation is a two-stream reward processing mechanism. Instead of a single reward signal, agents process positive and negative rewards independently. This is implemented by extending existing algorithms:
- Split Multi-Armed Bandit (MAB): Based on Human-Based Thompson Sampling (HBTS), where positive and negative streams are tracked via success and failure counts.
- Split Contextual Bandit (CB): Extends Contextual Thompson Sampling (CTS) by maintaining separate positive and negative reward streams, each with its own discount factors and weights.
- Split Reinforcement Learning (RL): Builds upon Split Q-Learning (SQL), using two independently updated Q-functions for positive and negative rewards.
A key aspect is the introduction of four hyper-parameters per stream (two discount factors and two weights) that allow for fine-tuning of how past rewards and current rewards influence decision-making. By adjusting these parameters, the authors can model various "clinically inspired" reward processing biases, such as those seen in addiction, ADHD, or Parkinson's disease.
Datasets
The paper evaluates their proposed framework across several tasks:
- Simulated tasks:
- A simple MAB task with bimodal reward distributions.
- A simulated MDP task with bimodal reward distributions.
- The classic Pac-Man game, with various stationarities (reward muting, scaling, and flipping) to simulate lifelong learning.
- Real-world dataset:
- The Iowa Gambling Task (IGT), a well-established benchmark for decision-making under uncertainty and risk.
No new datasets were introduced; instead, existing benchmarks were adapted to incorporate the two-stream reward processing.
Key Insights and Ideas
- Unified Framework: The "two-stream" approach successfully unifies MAB, CB, and RL problems under a single, flexible parametric framework. This is a significant step towards a more general model of sequential decision-making.
- Human-like Behavior: By modeling separate positive and negative reward streams and allowing for parameter tuning, the agents can exhibit diverse and clinically relevant behavioral trajectories that go beyond simple reward maximization. This is crucial for understanding and modeling human decision-making, especially in the context of disorders.
- Clinical Relevance: The framework's ability to mimic reward processing biases observed in various neurological and psychiatric disorders (e.g., addiction, ADHD, Parkinson's) is a major contribution. This opens doors for computational psychiatry and neuroscience.
- Flexibility is Key: The authors demonstrate that specific "biases" (parameter settings) can be beneficial in certain environments, leading to better performance than standard baselines. This challenges the notion that optimal agents must always be purely reward-maximizing.
- Lifelong Learning: The Pac-Man experiments highlight the framework's robustness in dynamic environments where reward structures change, suggesting its potential for lifelong learning agents.
Key Results
- Comparable Performance: In simulated tasks and the IGT, the split models (e.g., HBTS, SCTS, SQL) often performed comparably to, and sometimes better than, their standard non-split counterparts. For instance, in the MAB task, split models consistently showed advantages over other algorithms.
- "Mental" Agents' Performance: Several "mental" agents (e.g., PD, M, ADHD, CP) showed strong performance in specific tasks, demonstrating that tailored reward processing can be advantageous. For example, in the MAB task, PD and M agents were optimal for split MAB models.
- Pac-Man Success: Across various stationarities in Pac-Man, split models demonstrated competitive performance, with some "mental" agents even outperforming baselines in reward-flipping scenarios, aligning with clinical observations of ADHD.
- Faster Convergence: In some scenarios, split models converged more quickly than baselines, as visualized in learning curves (Figure 1 and 2).
Limitations and Future Work
- Model Simplification: The authors acknowledge that their "mental" models are simplified approximations of complex disorders and do not capture all aspects.
- Parameter Learning: The current work uses fixed parameter settings for the "mental" agents. Future work should focus on learning these parameters from actual patient data.
- Generalization: While the framework is unified, further exploration is needed to understand its performance across a wider range of tasks and environments.
- Multi-Agent Interactions: The authors suggest exploring multi-agent scenarios with agents exhibiting different reward processing biases.
- Optimal Parameter Discovery: Investigating how to find optimal parameters for specific tasks and criteria (e.g., longest survival vs. highest score) is a promising direction.
- Neuroscience Validation: Further tuning and extending the model to better align with modern neuroscience literature and testing on healthy subjects and patients are key future steps.
Summary updated