Even if suboptimal, novelty drives human exploration

Alireza Modirshanechi, Wei-Hsiang Lin, Xu He, Michael H. Herzog, Wulfram GerstnerbioRxiv 2022 · DOI

TL;DR

This paper investigates how humans explore complex environments, finding that novelty-seeking, rather than optimal information gain, drives exploration, even when it leads to suboptimal behavior in stochastic regions. This suggests humans employ computationally cheap, novelty-driven strategies for exploration, even if they aren't perfectly efficient.

What Problem Does This Paper Address?

Humans are adept at exploring their surroundings to find rewards, even when it involves navigating through many steps without immediate feedback. A key question in neuroscience and psychology is what internal "intrinsic" rewards guide this exploration. While theories suggest signals like novelty, surprise, or information gain act as intrinsic rewards, it's unclear which one truly drives human behavior. Furthermore, some intrinsic reward mechanisms can lead to a "noisy TV problem," where agents become attracted to irrelevant stochastic stimuli. This paper aims to determine if humans exhibit this attraction to stochasticity and, if so, which intrinsic reward signal is responsible. They tackle this by designing a multi-step decision-making task that disentangles different exploration strategies.

Methods

The researchers designed a novel experimental paradigm with 58 states and 3 goal states. Participants (and simulated RL agents) navigated this environment using actions that could be deterministic or lead to stochastic transitions. A key feature was a "stochastic part" of the environment, which was reward-free but highly stochastic, mimicking aspects of the "noisy TV problem."

They modeled human behavior using intrinsically motivated Reinforcement Learning (RL) agents. Four types of algorithms were tested:

  1. Novelty-seeking: Agents are guided by how infrequently they've visited states.
  2. Information-gain-seeking: Agents explore states that maximally update their understanding of the environment's structure.
  3. Surprise-seeking: Agents are drawn to unexpected state transitions.
  4. Exploration by optimistic initialization: A control condition with no intrinsic rewards, relying solely on optimistic initial estimates of rewards.

They used Bayesian model comparison and posterior predictive checks to determine which algorithm best explained human participants' choices.

Datasets

The study used data collected from 57 human participants performing the designed multi-step decision-making task. No external datasets were used for training or evaluation. The experimental paradigm itself served as the benchmark for evaluating the computational models.

Key Insights and Ideas

  • Humans are drawn to stochasticity, even when it's not rewarding: Participants persistently explored the stochastic, reward-free sub-region of the environment, even after finding the highest-value goal. This mirrors the "noisy TV problem" seen in some RL agents.
  • Novelty, not information gain, drives human exploration: Computational modeling revealed that novelty-seeking algorithms were the best fit for explaining human behavior, outperforming models driven by information gain or surprise. This suggests humans prioritize exploring novel states, even if it's not the most "optimal" strategy for maximizing information.
  • Reward optimism influences exploration extent, not strategy: While participants with higher "reward optimism" (expecting to find better rewards) explored more, their underlying exploration strategy remained novelty-driven.
  • Humans learn the environment's structure, but don't always act optimally: Participants were able to reconstruct the environment's map, demonstrating they understood its structure. However, their exploration in stochastic regions was suboptimal, suggesting a trade-off between computational cost and optimality.

Key Results

  • Participants spent less time in trap states and more time in the stochastic part of the environment over time.
  • Exploratory behavior (search duration and time in stochastic part) was negatively correlated with the reward value of the discovered goal.
  • The 2 CHF group (most optimistic) showed constant search duration, indicating persistent exploration of the stochastic part.
  • Novelty-seeking models provided the best fit for human participants' action choices, with a cross-validated accuracy of over 40% in the first episode, increasing to over 60% by the fifth episode.
  • Bayesian model comparison strongly favored novelty-seeking over information gain, surprise-seeking, and optimistic initialization.

Limitations and Future Work

  • Suboptimal exploration: The study highlights that human exploration, while guided by novelty, is not always theoretically optimal, especially in stochastic environments. This suggests a "resource-rational" approach where computationally cheaper strategies are favored.
  • Generalizability: The authors suggest further experimental studies are needed to investigate if these findings generalize to other types of human exploratory behavior, such as goal-directed exploration in different contexts or reward-free exploration strategies.
  • Definition of intrinsic rewards: The paper acknowledges that "novelty," "surprise," and "information gain" can have various precise mathematical definitions, and their findings are based on specific formulations. They expect their conclusions to be invariant to minor variations in these definitions.

Summary updated

Abstract

Abstract Humans successfully explore their environment to find ‘extrinsic’ rewards, even when exploration requires several intermediate reward-free decisions. It has been hypothesized that ‘intrinsic’ rewards such as novelty guide this reward-free exploration. However, different intrinsic rewards lead to different exploration strategies, some prone to suboptimal attraction to irrelevant stochastic stimuli, sometimes called the ‘noisy TV problem.’ Here, we ask whether humans show a similar attraction to reward-free stochasticity and, if so, which type of intrinsic reward guides their exploration. We design a multi-step decision-making paradigm where human participants search for rewarding states in an environment with a highly stochastic but reward-free sub-region. We show that (i) participants persistently explore the stochastic sub-region and (ii) their decisions are best explained by algorithms driven by novelty but not by ‘optimal’ algorithms driven by information gain. Our results suggest that humans use suboptimal but computationally cheap strategies for exploration in complex environments.
BibTeX
@misc{modirshanechi2022even,
  author = {Alireza Modirshanechi and Wei-Hsiang Lin and Xu He and Michael H. Herzog and Wulfram Gerstner},
  title = {{Even if suboptimal, novelty drives human exploration}},
  year = {2022},
  doi = {10.1101/2022.07.05.498835},
  howpublished = {bioRxiv},
}
Figure