Dynamics of Intrinsic Rewards for Autonomous Exploration

This direction is split into other directions: intrinsic reward decay. Its last version stays readable and citable here.

Version 1 of 162 papers68 conceptsfrozen 2026-10-04

In traditional reinforcement learning, an agent is a hedonist: it acts solely to maximize an external reward signal. However, in many real-world environments, these signals are "sparse"—a robot might wander for hours before accidentally stumbling upon a success. To bridge this gap, researchers have turned to Intrinsic Motivation (IM), a suite of techniques that provide agents with internal "pseudo-rewards" based on concepts like curiosity, novelty, and surprise.

The core intuition behind this research direction is that exploration should not be random. Instead, it should be driven by the agent's own learning progress. As Satinder Singh 0001, Andrew G. Barto, and Nuttapong Chentanez argued in their foundational work on Intrinsically Motivated Reinforcement Learning, an agent can autonomously build a hierarchy of skills by chasing "salient" events that it cannot yet predict.

The Mechanism of Curiosity

Most intrinsic rewards are derived from the failure of an agent's internal model. If an agent tries to predict the next state of the environment and fails, the resulting Prediction Error serves as a reward. This creates a feedback loop: the agent is attracted to areas where its model is weak, gathers data there, and improves its model.

Several recent methods refine how this error is measured:

  • Reconstruction Error: Toru Lin and Allan Jabri introduced MIMEx, which uses masked autoencoding. The agent is rewarded for visiting states where it cannot accurately "fill in the blanks" of its sensory input.
  • Motion Correspondence: Hsuan-Kung Yang and colleagues proposed using optical flow estimation errors to drive curiosity, which helps the agent focus on dynamic interactions rather than static noise.
  • Information Gaps: Alberto Caron and Vasilios Mavroudis emphasize Bayesian exploration, where rewards are tied to genuine "knowledge gaps" rather than environmental stochasticity.

The Functional Role of Boredom

A critical, often overlooked aspect of this research is the transience of the reward. As an agent masters a task, its prediction error drops to zero. In the framework of Jürgen Schmidhuber, this is the Formal Theory of Fun: "fun" is the first derivative of learning. When learning plateaus, the agent becomes "bored."

Boredom is not a bug; it is a vital regulatory mechanism. It forces the agent to transition from mastered behaviors to unexplored frontiers. Samuel J. Gershman frames this as subjective functions, where hedonic adaptation prevents an agent from getting trapped in high-reward but low-information states. Similarly, Mark H. Lee and Qinggang Meng describe a constraint-lifting mechanism where a robot "grows" its coordination by moving to new stages once current patterns become familiar.

The Vanishing Reward Trap

While boredom drives progress, it also creates the "Vanishing Reward" problem. If the intrinsic reward disappears too quickly—before the agent finds the external goal—the agent may stall.

Adrien Ecoffet, Jeff Clune, and their team identified two specific failure modes in Go-Explore:

  1. Detachment: An agent discovers a promising area but leaves it. By the time it wants to return, the curiosity reward for the path leading there has been "consumed," leaving the agent with no motivation to go back.
  2. Derailment: Standard exploration (like adding noise to actions) can prevent an agent from returning to a precise, difficult-to-reach state it previously discovered.

To combat this, researchers like Tianjun Zhang and Yi Wu developed NovelD, which rewards the difference in novelty between states, effectively pushing the agent toward the "frontier" of its knowledge. Others, such as Haozhe Ma, suggest Dual Random Networks to decouple the "newness" of a state from its potential "contribution" to the final goal.

Theoretical Convergence

The field is moving toward a unified understanding of these heuristics. Aly Lidayan and Stuart Russell 0001 have provided a theoretical bridge between intrinsic motivation and reward shaping. They show that by designing potential functions that eventually saturate, we can guarantee that an agent will eventually stop exploring and focus on the primary task, ensuring that curiosity serves as a scaffold for learning rather than a permanent distraction.

Papers in this version

Showing 60 of 62 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_dynamics_of_intrinsic_rewards_for_autono_9412d2c0,
  title = {Dynamics of Intrinsic Rewards for Autonomous Exploration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamics-of-intrinsic-rewards-for-autonomous-exploration/rdr_9412d2c0a8b445ec81675c050fa7dd7f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_dynamics_of_intrinsic_rewards_for_autono_9412d2c0_v1,
  title = {Dynamics of Intrinsic Rewards for Autonomous Exploration},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/dynamics-of-intrinsic-rewards-for-autonomous-exploration/rdr_9412d2c0a8b445ec81675c050fa7dd7f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}