An Empirical Study of Human Behavioral Agents in Bandits, Contextual Bandits and Reinforcement Learning

Baihan Lin, Guillermo Cecchi, Djallel Bouneffouf, Jenna Reinen, Irina Rish

TL;DR

This paper introduces a novel, flexible framework for sequential decision-making agents that models human behavior by splitting reward processing into two streams: positive and negative. This unified approach, inspired by neurological and psychiatric disorders, demonstrates improved learning and more nuanced behavioral trajectories across various decision-making tasks, from simple bandits to complex reinforcement learning environments.

What Problem Does This Paper Address?

Traditional artificial agents often learn to optimize for a single, consistent reward signal. However, human decision-making is far more complex, influenced by a variety of factors and often exhibiting different strategies that lead to similar outcomes. This paper addresses the gap in computational models by proposing a more realistic approach that captures the nuances of human reward processing. Specifically, it tackles the challenge of modeling how humans might differentially learn from positive and negative feedback, drawing inspiration from observed biases in reward processing associated with various neurological and psychiatric conditions. The goal is to create more flexible and potentially more effective agents by incorporating these "clinically-inspired" biases.

Methods

The core innovation is a two-stream reward processing mechanism. Instead of a single reward signal, the framework splits incoming rewards into positive and negative components. This allows for independent learning and weighting of each stream. The authors extend existing algorithms for Multi-Armed Bandits (MAB), Contextual Bandits (CB), and Reinforcement Learning (RL) to incorporate this two-stream approach.

  • Split MAB: Builds upon Human-Based Thompson Sampling (HBTS) by maintaining separate counts for successes and failures.
  • Split CB: Extends Contextual Thompson Sampling (CTS) by introducing parameters that control the influence of past positive and negative rewards, as well as the weight given to current positive and negative feedback.
  • Split RL: Adapts Split Q-Learning (SQL) by using two separate Q-functions, one for positive rewards and one for negative rewards, allowing for independent updates.

Crucially, the framework introduces four hyperparameters that can be tuned to mimic various reward-processing biases observed in conditions like addiction, ADHD, Alzheimer's, and chronic pain. This allows for a spectrum of agent behaviors, from standard agents (all parameters set to 1) to specialized "mental" agents.

Datasets

The paper evaluates its proposed models across several established and real-world datasets/environments:

  • Simulated tasks with particular reward distributions: Specifically, bimodal reward distributions in MAB and MDP tasks were used to test the agents' ability to handle complex reward landscapes.
  • Iowa Gambling Task (IGT): A real-world dataset capturing human decision-making under uncertainty with potential gains and losses.
  • PacMan game: Used to test agents in a more dynamic and complex environment with varying reward structures and stationarities.

No new datasets were introduced; instead, the focus was on adapting existing benchmarks to the two-stream reward processing paradigm.

Key Insights and Ideas

  • Human decision-making is not monolithic: The paper strongly suggests that modeling human behavior requires moving beyond a single reward signal to account for differential processing of positive and negative feedback.
  • "Mental" biases can be beneficial: Surprisingly, certain "clinically-inspired" reward processing biases, when modeled parametrically, can lead to improved performance in specific environments, especially those with non-stationary or adversarial characteristics. This highlights the adaptive nature of these biases in certain contexts.
  • Unified framework for diverse problems: The two-stream approach provides a flexible and unified way to model decision-making across MAB, CB, and RL, demonstrating its broad applicability.
  • Behavioral trajectories matter: The "mental" agents often exhibit distinct learning curves and action selection patterns that align with clinical observations, offering a computational bridge between AI and neuroscience.

Key Results

  • Improved learning speed: In several experiments, particularly in the Iowa Gambling Task and PacMan, the split models (e.g., SCTS, HBTS, SQL) converged faster and achieved higher cumulative rewards compared to their standard baselines. For instance, in IGT scheme 1, SCTS achieved a cumulative reward of 1200.76, significantly outperforming MAB baseline TS (991.26).
  • Competitive performance: Across various tasks and stationarities, the split models generally performed comparably to or better than standard algorithms.
  • "Mental" agents show promise: Specific "mental" agent configurations demonstrated superior performance in certain scenarios. For example, in the PacMan reward flipping scenario, ADHD-like agents showed particular advantage, and in the IGT, the "chronic pain" agent exhibited behaviors consistent with clinical observations.
  • Robustness: The split models were observed to be more robust, "never seem to fail catastrophically," maintaining an overall advantage over baselines.

Limitations and Future Work

  • Approximation of disorders: The authors explicitly state that the "mental" models are treated as "first approximations" and do not claim to fully capture all aspects of the corresponding disorders. The parameter settings are illustrative rather than derived from patient data.
  • Need for real-world data: Future work should focus on tuning these parameters using actual patient data to create more accurate and clinically relevant models.
  • Multi-agent interactions: Investigating how agents with different reward processing biases interact in multi-agent settings is a promising direction.
  • Exploration of other games and criteria: Extending the evaluation to a wider range of computer games and exploring different performance criteria (e.g., survival time vs. score) is suggested.
  • Interpretable humanoid systems: The discoveries could contribute to building more interpretable real-world humanoid decision-making systems.

Summary updated

Abstract

Artificial behavioral agents are often evaluated based on their consistent behaviors and performance to take sequential actions in an environment to maximize some notion of cumulative reward. However, human decision making in real life usually involves different strategies and behavioral trajectories that lead to the same empirical outcome. Motivated by clinical literature of a wide range of neurological and psychiatric disorders , we propose here a more general and flexible parametric framework for sequential decision making that involves a two-stream reward processing mechanism. We demonstrated that this framework is flexible and unified enough to incorporate a family of problems spanning multi-armed bandits (MAB), contextual bandits (CB) and reinforcement learning (RL), which decompose the sequential decision making process in different levels. Inspired by the known reward processing abnormalities of many mental disorders, our clinically-inspired agents demonstrated interesting behavioral trajectories and comparable performance on simulated tasks with particular reward distributions, a real-world dataset capturing human decision-making in gambling tasks, and the PacMan game across different reward stationarities in a lifelong learning setting.
BibTeX
@inproceedings{lin2020empirical,
  author = {Baihan Lin and Guillermo Cecchi and Djallel Bouneffouf and Jenna Reinen and Irina Rish},
  title = {{An Empirical Study of Human Behavioral Agents in Bandits, Contextual Bandits and Reinforcement Learning}},
  booktitle = {HAL},
  year = {2020},
}
Figure