Introduction
In the pursuit of Artificial General Intelligence, unsupervised reinforcement learning (RL) aims to discover useful behaviors without manually defined reward functions. Historically, this has been approached through "curiosity" or novelty-seeking, where agents are rewarded for exploring unknown states. However, in truly dynamic environments—those characterized by increasing entropy—novelty-seeking can be counterproductive. An agent seeking novelty in a chaotic world might find "bountiful novelty" in destructive events, such as falling off a cliff or being hit by a car.
Inspired by the biological principle of homeostasis, we propose Surprise Minimizing Reinforcement Learning (SMiRL). Instead of seeking novelty, SMiRL agents are motivated to maintain stable, predictable states amidst environmental chaos. This simple drive to minimize surprise leads to the emergence of complex survival behaviors.
The SMiRL Framework
The core of SMiRL is an unsupervised objective that treats "surprise" as a negative reward. The agent's goal is to maximize the probability of the states it visits under its own historical distribution.
Reward Function and Density Modeling
The agent receives a reward $r_t$ based on the log-probability of the current state $s_t$ under a density model $p_\theta$ trained on the agent's experience within its current "lifetime" (episode): $$r_t = \log p_{\theta}(s_t)$$
In simple environments, $p_\theta$ can be a basic distribution (e.g., Gaussian). In high-dimensional visual environments, a Variational Autoencoder (VAE) is used to model states in a compressed latent space.
Meta-Reasoning for Stability
As the density model $p_\theta$ updates with each new state, the agent must reason about how its current actions will influence its future beliefs. To achieve high long-term rewards, an agent might temporarily visit "surprising" states if they lead to a more stable and predictable environment later—a form of delayed gratification for the sake of stability.
Experimental Evaluation
SMiRL was evaluated in several "entropic" environments where inaction leads to failure or chaos.
- Tetris: In this naturally entropic world, blocks accumulate until the game ends. While SMiRL has no knowledge of the game's score, it learns to clear rows because an empty board is more predictable than a cluttered one. SMiRL achieved near-perfect play, matching oracle agents trained on the actual score.
- VizDoom: In scenarios like TakeCover, SMiRL agents learned to dodge fireballs to maintain a "boring," safe state, significantly outperforming curiosity-based baselines which were drawn toward the "novelty" of the fireballs.
- MuJoCo Humanoid: Robots placed on moving treadmills or pedestals learned to locomote and balance purely to avoid the "surprise" of falling off.
- HauntedHouse: In a navigation task with pursuing enemies, SMiRL agents learned to leave their starting area to find a "safe room," demonstrating that they will endure short-term surprise to reach long-term stability.
Discussion and Key Insights
SMiRL demonstrates that emergent intelligence can arise from the simple motive of stability. Unlike novelty-seeking methods (e.g., ICM or RND), SMiRL provides an "anti-curiosity" advantage in dangerous, dynamic worlds.
However, SMiRL faces the "static world" problem: in a perfectly stable environment, the agent may become a "couch potato," sitting still forever as that is the most predictable state. Its effectiveness relies on "prevailing sources of entropy" to drive behavior. Furthermore, the agent's success is highly dependent on the state representation; if the VAE fails to capture relevant features, the agent cannot effectively minimize surprise.
Future work includes using SMiRL as a "stability prior" to mitigate reward misspecification in supervised RL, ensuring that agents solve tasks while remaining within "normal," predictable bounds of behavior.
Summary updated