Abstract
Drawing an inspiration from behavioral studies of human decision making, we propose here a general parametric framework for a reinforcement learning problem, which extends the standard Q-learning approach to incorporate a two-stream framework of reward processing with biases biologically associated with several neurological and psychiatric conditions, including Parkinson's and Alzheimer's diseases, attention-deficit/hyperactivity disorder (ADHD), addiction, and chronic pain. For the AI community, the development of agents that react differently to different types of rewards can enable us to understand a wide spectrum of multi-agent interactions in complex real-world socioeconomic systems. Moreover, from the behavioral modeling perspective, our parametric framework can be viewed as a first step towards a unifying computational model capturing reward processing abnormalities across multiple mental conditions and user preferences in long-term recommendation systems.
Introduction
In standard Reinforcement Learning (RL), agents typically treat rewards as a single scalar value. However, biological systems—specifically the human brain—process gains and losses through different neural pathways and with varying sensitivities. This paper addresses the gap between traditional AI agents and the nuanced reward-processing observed in "computational psychiatry." For instance, patients with Parkinson’s disease often learn more effectively from negative consequences than positive rewards, while those with addictive behaviors may struggle to discount past rewards properly.
This research asks two key questions:
- Can we extend RL algorithms to mimic these specific human behavioral biases?
- Can these biologically-inspired, biased approaches actually outperform standard RL on specific tasks?
Methods: The Split Q Learning Framework
The authors introduce Split Q Learning (SQL), a parametric extension of the classic Q-Learning algorithm. Instead of a single Q-table, SQL maintains two distinct streams:
- $Q^+(s, a)$: Tracks accumulated positive feedback.
- $Q^-(s, a)$: Tracks accumulated negative feedback.
The Four Parameters ($\phi_1, \phi_2, \phi_3, \phi_4$)
The core innovation lies in how these tables are updated and combined using four weight parameters:
- $\phi_1$ and $\phi_3$: These act on the update rules for $Q^+$ and $Q^-$, respectively. They weight the "reward history" (memory) against new incoming rewards.
- $\phi_2$ and $\phi_4$: These are used during action selection. The agent chooses an action based on a weighted sum: $Q(s, a) = \phi_2 Q^+(s, a) + \phi_4 Q^-(s, a)$.
By adjusting these four values, the model can simulate different "personalities" or clinical conditions. Standard Q-Learning is revealed to be a specific, restricted case of SQL where all $\phi$ parameters are set to 1.
Parameter Discovery
The authors suggest using Inverse Reinforcement Learning (IRL) to extract these $\phi$ parameters from human behavioral data, effectively "reverse-engineering" a person's reward-processing bias. They also mention an adaptive version of SQL that uses the Gaussian Process Upper Confidence Bound (GP-UCB) algorithm to tune these parameters on the fly.
Experimental Setup
The researchers evaluated their agents across nine different computer game environments, including classic games (Pacman, Pong, Snake) and task-specific environments (Catcher, WaterWorld). These games were tested in both stationary (constant reward structures) and non-stationary environments (where the size and frequency of rewards change over time) to evaluate adaptation.
Key Insights and Results
- Performance Gains: Preliminary results showed that SQL agents outperformed classical Q-Learning in the long term under specific conditions, such as "positive-only" reward environments in Pacman.
- Adaptability: SQL agents behaved differently (and often more robustly) during transitions in non-stationary environments compared to standard agents.
- Clinical Modeling: The authors successfully mapped specific parameter ranges to various conditions. For instance, "Addiction" (AD) was modeled with high weights on positive history and selection, but lower sensitivity to negative updates.
- "Disorders" as Evolutionary Features: The paper leans into the perspective of evolutionary psychiatry, suggesting that traits associated with ADHD or anxiety aren't just "bugs"—they can be beneficial in specific environmental niches.
Limitations and Future Work
The authors acknowledge that their current parameter settings for disorders (like Alzheimer's or Chronic Pain) are "first approximations." Real-world reward processing is significantly more complex. Future work aims to validate the model against actual patient data, investigate "optimal" parameters for different success criteria, and explore how these biased agents interact in multi-agent systems.
Summary updated