Imitation as a model-free process in human reinforcement learning

Anis Najar, Emmanuelle Bonnet, Bahador Bahrami, Stefano PalminteribioRxiv 2019 · DOI

TL;DR

This paper investigates how humans integrate observed actions (imitation) into their reinforcement learning processes. Through computational modeling and experiments, they found that imitation is best explained by "value shaping," where observed actions directly influence the learner's value function, rather than biasing decisions or inferring the demonstrator's internal model. This suggests imitation is a model-free learning process, offering a simpler and more direct integration of social information.

What Problem Does This Paper Address?

Learning efficiently in complex environments often requires more than just trial-and-error with personal rewards. Social learning, particularly imitation, plays a crucial role in how humans acquire new skills and navigate social situations. However, the precise computational mechanisms by which imitation influences reinforcement learning (RL) remain unclear. Is it a temporary nudge to try a specific action, a complex inference of another's goals, or something else entirely? This paper aims to disentangle these possibilities by comparing three distinct computational models of imitation within an RL framework.

Methods

The researchers designed a novel social reinforcement learning task where participants (learners) could observe the choices of another individual (demonstrator) before making their own decisions. They then compared three computational models to explain the observed behavior:

  1. Decision Biasing: This model suggests that observing a demonstrator's action simply biases the learner's next action choice, without altering their underlying value estimates for actions.
  2. Model-Based Imitation: This approach posits that the learner infers the demonstrator's internal "value function" (what the demonstrator prefers) using inverse reinforcement learning and then uses this inferred model to guide their own actions.
  3. Value Shaping: This hypothesis proposes that the demonstrator's actions directly modify the learner's own value function, effectively "shaping" their preferences without requiring the learner to build a model of the demonstrator.

They tested these models against empirical data from two experiments using model comparison techniques, including AIC (Akaike Information Criterion) and Bayesian model selection, to determine which model best explained the participants' behavior.

Datasets

The study used data collected from two experiments with human participants.

  • Experiment 1: N=24 healthy participants.
  • Experiment 2: N=44 healthy participants.

The experiments involved a probabilistic instrumental learning task where participants made binary choices between abstract visual stimuli that yielded rewards. The key manipulation was the presence and quality of demonstrations from a "demonstrator" (controlled by the experimenter). The demonstrator was either "Skilled" (80% correct choices) or "Unskilled" (20% correct choices). The quality of the demonstrator was manipulated within-subject in Experiment 1 and between-subject in Experiment 2.

Key Insights and Ideas

  • Imitation is Model-Free: The most striking finding is that imitation, when adaptive (i.e., when the demonstrator is skilled), operates as a model-free process. This means learners don't necessarily need to infer the demonstrator's goals or internal state; they directly update their own value estimates based on observed actions.
  • Value Shaping Dominates: The "value shaping" model consistently outperformed other hypotheses, suggesting that observed actions directly influence the learner's value function, making those actions more or less attractive.
  • Adaptive Imitation: Humans are not passive imitators. They can infer the skill of a demonstrator and adjust their imitation accordingly, down-regulating imitation when the demonstrator is unskilled. This suggests a meta-learning process where the learner learns how to imitate.
  • Egocentric Bias: Learners tend to overweight their own experienced rewards compared to information gained from observing others, even when the demonstrator is skilled. This "egocentric bias" suggests a preference for self-generated feedback.

Key Results

  • Model Comparison: The value shaping (VS) model was the best fit for the data in both experiments, particularly when the demonstrator was skilled. In contrast, when the demonstrator was unskilled, a baseline Rescorla-Wagner model (RW) that ignores social information was often the best fit, indicating that imitation is suppressed when it's not beneficial.
  • Parameter Analysis:
    • Imitation learning rates were significantly higher when observing a Skilled Demonstrator compared to an Unskilled one.
  • Private reward learning rates were consistently higher than imitation learning rates, supporting the egocentric bias.
  • The adaptive modulation of imitation was specific to the imitation learning rate, with other parameters (like choice temperature) unaffected by demonstrator skill.

Limitations and Future Work

  • Task Specificity: The study used a specific type of social reinforcement learning task. Future work could explore whether these findings generalize to more complex, multi-step decision-making scenarios.
  • Distinguishing Shaping Methods: The paper notes that in single-step problems, distinguishing between reward shaping, value shaping, and policy shaping is difficult. Further research in multi-step paradigms is needed to clarify these distinctions.
  • Clinical Implications: The authors suggest that understanding the computational mechanisms of imitation is crucial for understanding social influence and could have implications for psychiatric conditions where social cognition is impaired. Future research could investigate how these computational impairments manifest in conditions like depression, anxiety, or autism.
  • "Subversive" Imitation: The paper raises the intriguing idea that value shaping might lead to a deeper integration of others' preferences, potentially leading to a "subversive" adoption of others' values. This warrants further investigation.

Summary updated

Abstract

While there is not doubt that social signals affect human reinforcement learning, there is still no consensus about their exact computational implementation. To address this issue, we compared three hypotheses about the algorithmic implementation of imitation in human reinforcement learning. A first hypothesis, decision biasing , postulates that imitation consists in transiently biasing the learner’s action selection without affecting her value function. According to the second hypothesis, model-based imitation , the learner infers the demonstrator’s value function through inverse reinforcement learning and uses it for action selection. Finally, according to the third hypothesis, value shaping , demonstrator’s actions directly affect the learner’s value function. We tested these three psychologically plausible hypotheses in two separate experiments (N = 24 and N = 44) featuring a new variant of a social reinforcement learning task, where we manipulated the quantity and the quality of the demonstrator’s choices. We show through model comparison that value shaping is favored, which provides a new perspective on how imitation is integrated into human reinforcement learning.
BibTeX
@misc{najar2019imitation,
  author = {Anis Najar and Emmanuelle Bonnet and Bahador Bahrami and Stefano Palminteri},
  title = {{Imitation as a model-free process in human reinforcement learning}},
  year = {2019},
  doi = {10.1101/797407},
  howpublished = {bioRxiv},
}
Figure