Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL

Baihan Lin, Guillermo A. Cecchi, Djallel Bouneffouf, Jenna Reinen, Irina Rish

TL;DR

This paper introduces a novel, flexible framework for sequential decision-making agents that models reward processing as two independent streams: one for positive rewards and one for negative rewards. This "split" mechanism allows for a more nuanced representation of human behavior, particularly in the context of neurological and psychiatric disorders, and unifies models across Multi-Armed Bandits (MAB), Contextual Bandits (CB), and Reinforcement Learning (RL).

What Problem Does This Paper Address?

Traditional artificial agents often assume a single, consistent reward processing mechanism. However, human decision-making is far more complex, with individuals exhibiting diverse strategies and behavioral patterns that can lead to similar outcomes. This variability is particularly evident in individuals with neurological and psychiatric disorders, where reward processing is known to be disrupted. The authors aim to bridge this gap by developing a more flexible agent framework that can capture these diverse reward processing biases, moving beyond the standard "one-size-fits-all" approach. This is crucial for building more realistic and interpretable AI systems, as well as for gaining deeper insights into human cognition and its disorders.

Methods

The core innovation is a two-stream reward processing mechanism. Instead of a single reward signal, agents process positive and negative rewards independently. This is implemented by extending existing algorithms for MAB, CB, and RL:

  • Split Multi-Armed Bandit (MAB): Builds upon Human-Based Thompson Sampling (HBTS) by maintaining separate success and failure counts for positive and negative rewards.
  • Split Contextual Bandit (CB): Extends Contextual Thompson Sampling (CTS) by using two independent streams for positive and negative reward updates, incorporating discount factors and weights for each stream.
  • Split Reinforcement Learning (RL): Adapts Split Q-Learning (SQL) by maintaining two independent Q-functions, one for positive rewards ($Q^+$) and one for negative rewards ($Q^-$).

The framework introduces four key hyperparameters per stream: $\lambda^+$ and $\lambda^-$ (discount factors for past rewards) and $w^+$ and $w^-$ (weights for current rewards). By tuning these parameters, the authors can model various reward processing biases observed in different mental disorders.

Datasets

The paper evaluates its proposed models on a combination of simulated environments and a real-world dataset:

  • Simulated MAB/MDP tasks: Used to test agents with bimodal reward distributions, where rewards can be positive or negative.
  • Iowa Gambling Task (IGT): A real-world dataset that simulates decision-making under uncertainty with potentially disadvantageous choices. The paper uses two schemes of the IGT.
  • PacMan game: Used to evaluate agents in a more complex, dynamic environment with various reward stationarities (e.g., stochastic reward muting, scaling, and flipping).

Key Insights and Ideas

  • Unified Framework for Diverse Problems: The split reward mechanism provides a single, flexible parametric framework that can be applied to MAB, CB, and RL problems, demonstrating a unifying principle in sequential decision-making.
  • Modeling Human-like Biases: By tuning the $\lambda$ and $w$ parameters, the model can effectively capture a range of reward processing biases observed in individuals with various neurological and psychiatric disorders (e.g., addiction, ADHD, Parkinson's). This offers a computational lens for studying these conditions.
  • Behavioral Trajectories Matter: The paper emphasizes that agents can achieve comparable final performance through very different behavioral trajectories, mirroring human decision-making. The "mental" agents often exhibit unique learning curves and strategies that align with clinical observations.
  • Biases Can Be Beneficial: Counter-intuitively, certain "disorder-inspired" biases can lead to superior performance in specific environments, especially those with non-stationary or adversarial conditions. This highlights the adaptive nature of these biases in certain contexts.

Key Results

  • Competitive Performance: Across MAB, CB, and RL tasks, the proposed split models (HBTS, SCTS, SQL) consistently performed competitively with, and often outperformed, their standard baselines. For instance, in the MAB task, HBTS showed a winning rate of 52.65% over non-split TS.
  • Iowa Gambling Task Success: In the IGT, the split contextual bandit (SCTS) achieved a significantly higher cumulative reward (1200.76) than baselines like TS (991.26), LinUCB (1165.23), and QL (1086.33) in scheme 1.
  • PacMan Adaptability: In the PacMan game, split models demonstrated competitive performance across various stationarities, even in complex environments where state-less CB agents were not ideally suited.
  • Clinically Inspired Models Show Promise: Several "mental" agents, such as those inspired by Parkinson's (PD) and chronic pain (CP), showed strong performance in specific tasks, aligning with known clinical observations about their reward processing. For example, the PD agent was better at learning to avoid negative outcomes.

Limitations and Future Work

  • Model Simplification: The authors acknowledge that their "mental" models are simplified representations and do not capture the full complexity of actual disorders.
  • Parameter Learning: While the paper demonstrates the flexibility of parameter tuning, future work could focus on learning these parameters from actual patient data.
  • Generalization and Real-World Application: Further research is needed to explore optimal parameter settings in a wider range of computer games and real-world scenarios, and to test the model's ability to capture more nuanced clinical observations.
  • Multi-Agent Interactions: Investigating how agents with different reward processing biases interact in multi-agent settings is a promising future direction.
  • Theoretical Guarantees: While empirical results are strong, further theoretical analysis of the split models, particularly for non-Bernoulli bandits, could be beneficial.

Summary updated

Abstract

Artificial behavioral agents are often evaluated based on their consistent behaviors and performance to take sequential actions in an environment to maximize some notion of cumulative reward. However, human decision making in real life usually involves different strategies and behavioral trajectories that lead to the same empirical outcome. Motivated by clinical literature of a wide range of neurological and psychiatric disorders, we propose here a more general and flexible parametric framework for sequential decision making that involves a two-stream reward processing mechanism. We demonstrated that this framework is flexible and unified enough to incorporate a family of problems spanning multi-armed bandits (MAB), contextual bandits (CB) and reinforcement learning (RL), which decompose the sequential decision making process in different levels. Inspired by the known reward processing abnormalities of many mental disorders, our clinically-inspired agents demonstrated interesting behavioral trajectories and comparable performance on simulated tasks with particular reward distributions, a real-world dataset capturing human decision-making in gambling tasks, and the PacMan game across different reward stationarities in a lifelong learning setting.
BibTeX
@inproceedings{lin2020unified,
  author = {Baihan Lin and Guillermo A. Cecchi and Djallel Bouneffouf and Jenna Reinen and Irina Rish},
  title = {{Unified Models of Human Behavioral Agents in Bandits, Contextual Bandits and RL}},
  booktitle = {HAL},
  year = {2020},
}
Figure