In reinforcement learning, intrinsic motivation acts as a critical catalyst for discovery in environments where external rewards are sparse or non-existent. By rewarding an agent for encountering novelty, experiencing surprise, or making learning progress, we encourage it to explore its surroundings without explicit human guidance.
However, a fundamental mathematical reality underlies almost all curiosity mechanisms: intrinsic rewards are inherently transient. As an agent interacts with its environment, its predictive models improve, states become familiar, and the once-potent exploration bonus decays toward zero.
This phenomenon of intrinsic reward decay—often conceptualized as algorithmic "boredom"—is a double-edged sword. When properly managed, it provides a elegant mathematical mechanism for transitioning an agent from exploration to exploitation. When unmanaged, it leads to premature stagnation, policy degeneration, and a complete failure to navigate complex environments.
The Theoretical Necessity of Boredom
Why do we want intrinsic rewards to decay? If an exploration bonus remained permanently high everywhere, an agent would be perpetually distracted, endlessly revisiting known states or getting trapped by environmental noise.
As formalized by Schmidhuber, boredom is a critical functional mechanism defined as the first derivative of learning progress. When an agent's ability to compress or predict a specific data sequence reaches a plateau, its learning progress drops to zero. To maximize its total reward, the agent must experience a corresponding drop in intrinsic motivation, forcing it to abandon mastered behaviors and seek out more complex, uncompressed features. This perspective is mirrored in early hierarchical frameworks by Singh et al., where tying intrinsic rewards to prediction errors naturally simulates boredom, shifting the agent's focus from mastered skills to unexplored domains.
From an optimization standpoint, this decay ensures asymptotic consistency—the guarantee that an agent will eventually stop exploring and focus entirely on the main task objective. Caron et al. demonstrate that information-theoretic exploration bonuses naturally contract as an agent's model of the environment improves, ensuring that intrinsic motivation safely diminishes as true knowledge is gained. Similarly, Zhang et al. show that designing rewards based on novelty gradients allows the exploration signal to equalize across states once an environment is fully mapped, smoothly transitioning the policy to external task rewards without requiring manual reward scheduling.
The Failure Modes of Premature Decay
While decay is theoretically elegant, practical implementations frequently suffer from premature convergence, where the exploration drive vanishes long before the agent discovers any external goals. This failure manifests in several distinct ways:
- Rapid Model Convergence: If an auxiliary task or predictive model is too simple, it learns too quickly. Achiam et al. and Lin et al. highlight that when a curiosity module masters environment transitions or input reconstruction prematurely, the intrinsic reward disappears, leaving the agent stranded in a sparse-reward desert without the motivation to continue searching. This issue is compounded if the primary value function itself converges too quickly, halting the propagation of reward signals across the state space, as noted by Simmons-Edler et al..
- The Physics Plateau: Raileanu et al. observe that prediction-error methods often vanish once an agent learns the basic physics of its environment. The agent becomes "bored" by simple movements before it ever discovers complex object interactions or distant goals. This pattern of high initial engagement followed by a rapid plateau is a common bottleneck in open-ended investigative tasks, as documented by Yin et al..
- Policy Degeneration: When an environment becomes familiar and intrinsic rewards drop to zero, the agent's policy can actively degenerate. As observed by Pathak et al. and Taiga et al., without a persistent extrinsic goal or a mechanism for skill retention, the agent loses the motivation to even maintain its acquired behaviors once the curiosity signal collapses.
The Propagation and Frontier Bottlenecks
The transience of intrinsic rewards introduces severe algorithmic challenges for standard reinforcement learning frameworks, particularly regarding how rewards are distributed over time and space.
The Propagation Problem
Standard temporal difference (TD) learning relies on propagating reward signals backward through time to build a value function. However, as Ostrovski et al. point out, novelty bonuses diminish immediately upon visitation. Because these exploration signals are short-lived, standard TD learning often fails to propagate them across long time horizons quickly enough for the agent to form a coherent, long-term exploration policy. Conversely, Shyam notes that additive novelty bonuses can sometimes cause the opposite problem: over-commitment, where an agent continues to haunt a specific area because the exploration reward is not unlearned quickly enough to reflect the agent's current knowledge.
The Frontier Problem
To reach a distant, unexplored frontier, an agent must repeatedly traverse known, familiar paths. But standard curiosity methods cause the agent to lose interest in a state after a single encounter, regardless of its strategic importance. As argued by Ma et al., this premature decrease in motivation prevents thorough investigation of paths critical for long-term success.
[Agent] ----> [Known Path] ----> [Known Path] ----> [Unexplored Frontier]
| |
Novelty Drops Novelty Drops
Reward Decays Reward Decays
| |
v v
(Agent turns back / Exploration stalls)
This vanishing novelty along well-trodden paths effectively stalls long-term discovery. The agent refuses to cross known territory to reach distant frontiers, a limitation highlighted across various environments by Badia et al., Bigazzi et al., Li et al., and Liu et al..
Engineering Sustainable Exploration
To combat premature decay while preserving the benefits of asymptotic convergence, researchers have developed several sophisticated strategies to regulate, sustain, or restructure intrinsic rewards.
Renewable Curiosity
In lifelong learning or task-agnostic settings, Parisi et al. propose treating curiosity as a renewable resource. By introducing stochastic or periodic resets of novelty metrics, they prevent the permanent decay of intrinsic motivation. Refreshing these counts maintains a persistent exploration pressure and prevents the vanishing of policy gradients over long durations.
Controlled Decay Factors
Rather than letting rewards drop arbitrarily based on neural network dynamics, Mavrin et al. scale exploration bonuses using a decay factor explicitly derived from the theoretical convergence rate of quantile estimation. This mechanism ensures that exploration incentives diminish systematically as the agent gains experience with a specific state-action pair, preventing permanent attraction to environmental noise while preserving structured curiosity.
Representation Contraction
Erraqabi et al. introduce an alternative approach by incorporating a contraction term into representation learning. This term actively reduces the distance in representation space between states connected by successfully executed skills. As these paths compress, the intrinsic reward for visiting them naturally decreases, creating a controlled boredom effect that systematically drives the agent's exploration policy outward toward the uncompressed frontier.
Non-Additive Marginal Gains
Recognizing that standard additive rewards fail to capture the nuances of diminishing returns, Prajapat frames exploration through submodular reinforcement learning. By explicitly optimizing for history-dependent marginal gains rather than cumulative novelty, the agent naturally accounts for saturation. This prevents it from getting trapped in high-reward regions whose informational value has already been exhausted, offering a mathematically rigorous solution to the problem of exploration stagnation.
Key Foundational Work
Intrinsically Motivated Reinforcement Learning (2004) — contributed 1 concept elements to this direction.
Learning sensory-motor cortical mappings without training (1998) — contributed 1 concept elements to this direction.