Merits of curiosity: a simulation study

Lucas Gruaz, Alireza Modirshanechi, Johanni Brea

TL;DR

Curiosity isn't a one-size-fits-all mechanism. By simulating agents across a diverse "zoo" of environments, researchers found that the "best" way to be curious depends entirely on your environment and your goals. However, a hybrid strategy combining novelty-seeking and information gain acts as a robust "Swiss Army knife," achieving near-optimal performance across almost all scenarios.

What Problem Does This Paper Address?

In neuroscience and AI, curiosity is often framed as an "intrinsic reward"—an internal signal that encourages exploration even when there’s no immediate external payoff (like food or money). But the field is crowded with different mathematical definitions of curiosity: Should an agent seek out novel states it hasn't seen often? Should it seek surprising transitions? Or should it aim for information gain to better understand the world's rules?

The problem is that different studies often reach different conclusions because they use different environments and different definitions of "success." This paper seeks to demystify this "Environment-Reward-Performance" triangle. The authors ask: How does the structure of the world (e.g., having "sink" states you can't escape) change which curiosity drive is most effective?

Methods

The researchers conducted a systematic simulation study using a model-based Reinforcement Learning (RL) framework.

  1. Environment Generation: They developed a novel algorithm to generate diverse, multi-step environments. It starts with a branching maze, integrates grid-like "rooms," and assigns specific properties:
  2. Sinks: Easy to enter, hard to leave (like social media "doom-scrolling").
  3. Sources: High-connectivity hubs (like an airport).
  4. Stochasticity: Randomness in transitions (like a weather forecast).
  5. The Agents: They simulated agents driven by six representative intrinsic rewards:
  6. Novelty: Rewards visiting rarely seen states.
  7. Surprise: Rewards experiencing unexpected transitions.
  8. Information Gain (IG): Rewards actions that reduce uncertainty about the world's model.
  9. Empowerment: Rewards reaching states where the agent has the most control.
  10. MOP & SPIE: Modern, complex objectives focused on occupancy and bottleneck states.
  11. The Objectives: They measured performance based on State Discovery (finding everything), Model Accuracy (learning the rules), and Uniform Visitation (not ignoring any part of the world).

Datasets

The paper does not use external datasets. Instead, it introduces a procedural environment generator that created 135 unique environment variants for evaluation. This benchmark is a significant contribution, as it allows researchers to test curiosity in a standardized yet diverse way that mimics real-world complexities like bottlenecks and "trap" states.

Key Insights and Ideas

  • Context is King: There is no "optimal" curiosity. If your goal is to map the world (discovery), Information Gain is your best bet. If you need to keep a balanced eye on everything (uniform visitation), Novelty wins.
  • The "Two Axes" of Curiosity: The authors found that Novelty and Information Gain are the two fundamental pillars of exploration. While they are mathematically distinct, they are the most robust.
  • The Empowerment Trap: "Naive" empowerment—seeking states where you have the most control—actually hinders exploration. Because the agent doesn't know the rules of unknown regions, it perceives them as having "low control" and avoids them, getting stuck in familiar territory.
  • The Hybrid Advantage: A "hybrid" agent that combines Information Gain and Novelty (weighting IG at about 75%) performed near-optimally in every environment and objective. It also excelled at finding "extrinsic" rewards (like hidden goals) faster than specialized agents.

Key Results

  • Information Gain consistently led to the lowest error in the agent's internal model of the world.
  • Novelty-seeking agents were the most successful at avoiding "sinks" and ensuring they didn't spend too much time in one place.
  • In Sink environments, novelty-seeking agents discovered states significantly faster than surprise-seeking agents, who often got "trapped" trying to resolve the randomness of the sink.
  • The Hybrid Agent achieved a normalized performance score close to 0 (where 0 is best and 1 is worst) across all 135 environment variants, proving its generalizability.

Limitations and Future Work

The study primarily uses tabular RL, meaning the environments have discrete, countable states. While this is perfect for a controlled simulation, real-world environments (and high-end AI) often deal with continuous, high-dimensional spaces where "states" aren't as clearly defined.

The authors also assume perfect memory (the agent remembers every transition it has ever made). Future work could explore how limited memory or "forgetting" affects these curiosity drives. Finally, the environments were static; testing how these drives handle a world that changes over time (non-stationary dynamics) is a promising next step.

Summary updated

Abstract

'Why are we curious?' has been among the central puzzles of neuroscience and psychology in the past decades. A popular hypothesis is that curiosity is driven by intrinsically generated reward signals, which have evolved to support survival in complex environments. To formalize and test this hypothesis, we need to understand the enigmatic relationship between (i) intrinsic rewards (as drives of curiosity), (ii) optimality conditions (as objectives of curiosity), and (iii) environment structures. Here, we demystify this relationship through a systematic simulation study. First, we propose an algorithm to generate environments that capture key abstract features of different real-world situations. Then, we simulate artificial agents that explore these environments by seeking one of six representative intrinsic rewards: novelty, surprise, information gain, empowerment, MOP, and SPIE. Finally, we evaluate the exploration performance of these simulated agents regarding three potential objectives of curiosity: state discovery, model accuracy, and uniform state visitation. Our results show that the comparative performance of each intrinsic reward is highly dependent on the environmental features and the curiosity objective; this indicates that 'optimality' in top-down theories of curiosity needs a precise formulation of assumptions. Nevertheless, we found that agents seeking a combination of novelty and information gain always achieve a close-to-optimal performance. This suggests that novelty and information gain are two principal axes of curiosity-driven behavior. These results pave the way for the further development of computational models of curiosity and the design of theory-informed experimental paradigms.
BibTeX
@misc{gruaz2024merits,
  author = {Lucas Gruaz and Alireza Modirshanechi and Johanni Brea and Johanni Brea},
  title = {{Merits of curiosity: a simulation study}},
  year = {2024},
  doi = {10.31234/osf.io/evm9n},
  howpublished = {PsyArXiv},
}
Figure