Abstract
Split Q Learning (SQL) introduces a reinforcement learning framework that decouples positive and negative rewards into separate processing streams. Inspired by reward-processing biases observed in neurological and psychiatric disorders—such as ADHD, Parkinson’s, and addiction—SQL utilizes four tunable parameters to weight reward history and current feedback. This architecture allows for the simulation of clinical profiles and demonstrates improved adaptability and performance in non-stationary environments compared to standard Q-learning.
Introduction
Standard Reinforcement Learning (RL) algorithms typically collapse rewards into a single scalar value. However, biological systems process "carrots" and "sticks" through distinct neural pathways. In computational psychiatry, specific mental conditions are characterized by abnormalities in these pathways. For example, individuals with Parkinson’s disease may exhibit a heightened sensitivity to negative feedback, while those with addictive behaviors may over-weight past high-value rewards. This paper explores whether extending RL algorithms to incorporate these human-like biases can improve agent performance in complex tasks.
The Split Q Learning Framework
The authors propose Split Q Learning (SQL), which maintains two separate Q-tables: $Q^+$ for positive feedback and $Q^-$ for negative feedback. The final decision-making value $Q(s, a)$ is derived using four weight parameters ($\phi_1$ through $\phi_4$):
- $\phi_1$ and $\phi_3$: These weight the "memory" or history of previously accumulated positive and negative rewards, respectively.
- $\phi_2$ and $\phi_4$: These weight the current incoming positive and negative rewards.
By adjusting these parameters, the researchers simulate various "disorder profiles." As shown in Table 1, a high $\phi_4$ (e.g., $100 \pm 10$) simulates the negative reward sensitivity of Parkinson’s (PD), while reduced weights on reward history ($\phi_1, \phi_3$) can reflect the shorter attention spans or faster-switching focus associated with ADHD.
Experimental Evaluation
SQL was evaluated across nine game environments, including classic titles like Pacman and Pong, and simulated environments like WaterWorld. These were tested in both stationary and non-stationary conditions, where reward scales were shifted mid-game to test adaptability. An adaptive version of SQL using Gaussian Process Upper Confidence Bound (GP-UCB) was also implemented to tune parameters dynamically as the environment changed.
Results and Discussion
- Performance and Adaptability: SQL outperformed classical Q-learning in the long term, particularly in "positive-only" reward environments. SQL agents also showed superior adaptability during transitions between reward regimes compared to standard agents.
- The Utility of Bias: The study suggests that "biased" learning is not inherently disadvantageous. Traits associated with mental disorders may represent evolutionary adaptations for specific environmental niches, such as hyper-vigilance or rapid attention switching.
- Clinical Mapping: The model successfully mapped parameter configurations to clinical observations, providing a computational bridge between AI and neuroscience. This suggests RL agents can serve as models to understand reward disruptions in conditions like Alzheimer’s and chronic pain.
Conclusion and Future Work
SQL demonstrates that splitting reward signals provides a powerful mechanism for controlling agent behavior and modeling biological systems. While current parameter settings are first approximations, future work will focus on using Inverse Reinforcement Learning (IRL) to extract $\phi$ parameters directly from human behavioral data. Additionally, exploring multi-agent interactions between differently biased agents could provide insights into complex socioeconomic systems.
Summary updated