TL;DR
Curiosity isn't a one-size-fits-all mechanism. By simulating agents across a diverse "zoo" of environments, researchers found that the "best" way to be curious depends entirely on your environment and your goals. However, a hybrid strategy combining novelty-seeking and information gain acts as a robust "Swiss Army knife," achieving near-optimal performance across almost all scenarios.
What Problem Does This Paper Address?
In neuroscience and AI, curiosity is often framed as an "intrinsic reward"—an internal signal that encourages exploration even when there’s no immediate external payoff (like food or money). But the field is crowded with different mathematical definitions of curiosity: Should an agent seek out novel states it hasn't seen often? Should it seek surprising transitions? Or should it aim for information gain to better understand the world's rules?
The problem is that different studies often reach different conclusions because they use different environments and different definitions of "success." This paper seeks to demystify this "Environment-Reward-Performance" triangle. The authors ask: How does the structure of the world (e.g., having "sink" states you can't escape) change which curiosity drive is most effective?
Methods
The researchers conducted a systematic simulation study using a model-based Reinforcement Learning (RL) framework.
- Environment Generation: They developed a novel algorithm to generate diverse, multi-step environments. It starts with a branching maze, integrates grid-like "rooms," and assigns specific properties:
- Sinks: Easy to enter, hard to leave (like social media "doom-scrolling").
- Sources: High-connectivity hubs (like an airport).
- Stochasticity: Randomness in transitions (like a weather forecast).
- The Agents: They simulated agents driven by six representative intrinsic rewards:
- Novelty: Rewards visiting rarely seen states.
- Surprise: Rewards experiencing unexpected transitions.
- Information Gain (IG): Rewards actions that reduce uncertainty about the world's model.
- Empowerment: Rewards reaching states where the agent has the most control.
- MOP & SPIE: Modern, complex objectives focused on occupancy and bottleneck states.
- The Objectives: They measured performance based on State Discovery (finding everything), Model Accuracy (learning the rules), and Uniform Visitation (not ignoring any part of the world).
Datasets
The paper does not use external datasets. Instead, it introduces a procedural environment generator that created 135 unique environment variants for evaluation. This benchmark is a significant contribution, as it allows researchers to test curiosity in a standardized yet diverse way that mimics real-world complexities like bottlenecks and "trap" states.
Key Insights and Ideas
- Context is King: There is no "optimal" curiosity. If your goal is to map the world (discovery), Information Gain is your best bet. If you need to keep a balanced eye on everything (uniform visitation), Novelty wins.
- The "Two Axes" of Curiosity: The authors found that Novelty and Information Gain are the two fundamental pillars of exploration. While they are mathematically distinct, they are the most robust.
- The Empowerment Trap: "Naive" empowerment—seeking states where you have the most control—actually hinders exploration. Because the agent doesn't know the rules of unknown regions, it perceives them as having "low control" and avoids them, getting stuck in familiar territory.
- The Hybrid Advantage: A "hybrid" agent that combines Information Gain and Novelty (weighting IG at about 75%) performed near-optimally in every environment and objective. It also excelled at finding "extrinsic" rewards (like hidden goals) faster than specialized agents.
Key Results
- Information Gain consistently led to the lowest error in the agent's internal model of the world.
- Novelty-seeking agents were the most successful at avoiding "sinks" and ensuring they didn't spend too much time in one place.
- In Sink environments, novelty-seeking agents discovered states significantly faster than surprise-seeking agents, who often got "trapped" trying to resolve the randomness of the sink.
- The Hybrid Agent achieved a normalized performance score close to 0 (where 0 is best and 1 is worst) across all 135 environment variants, proving its generalizability.
Limitations and Future Work
The study primarily uses tabular RL, meaning the environments have discrete, countable states. While this is perfect for a controlled simulation, real-world environments (and high-end AI) often deal with continuous, high-dimensional spaces where "states" aren't as clearly defined.
The authors also assume perfect memory (the agent remembers every transition it has ever made). Future work could explore how limited memory or "forgetting" affects these curiosity drives. Finally, the environments were static; testing how these drives handle a world that changes over time (non-stationary dynamics) is a promising next step.
Summary updated