Introduction
In the field of Unsupervised Reinforcement Learning (URL), agents are tasked with learning meaningful behaviors in environments without the guidance of external reward signals. To navigate these environments, agents rely on intrinsic motivation. Historically, research has diverged into two primary strategies: Surprise-Maximization (Curiosity), where agents seek novel and unpredictable states, and Surprise-Minimization (Empowerment), where agents seek to make their environment predictable and stable.
Both strategies possess inherent flaws. Curiosity-driven agents often fall victim to the "Noisy TV" problem, becoming trapped by irreducible randomness. Conversely, stability-seeking agents may succumb to the "Dark Room" scenario, where they remain stationary to avoid any new information. This paper introduces S-Adapt, an unsupervised agent designed to autonomously decide whether to seek or avoid surprise based on the specific demands of its environment.
Methodology: The S-Adapt Framework
S-Adapt treats the selection of an intrinsic motivation objective as a Multi-Armed Bandit (MAB) problem. At the beginning of each episode, the agent selects between two "arms":
- Maximize Surprise ($S_{max}$): Encourages exploration of novel states.
- Minimize Surprise ($S_{min}$): Encourages the stabilization of the environment.
Surprise is quantified as the negative log-likelihood of a state under the agent's historical state distribution.
The Controllability Metric
The core innovation of S-Adapt is its feedback mechanism for the bandit. To determine which objective is more effective, the authors introduce a metric based on controllability. The bandit is rewarded based on the absolute percent difference between the entropy of the agent's state distribution and the entropy of a random agent.
The underlying logic is that if an agent can make its experience significantly more chaotic or significantly more orderly than a random walk, it is successfully exerting control over the environment. The bandit naturally favors the objective that results in the largest deviation from baseline randomness.
Architecture
The system utilizes a Deep Q-Network (DQN). To allow the agent to track its own entropy over time, the input is augmented with "sufficient statistics," specifically the mean and variance of visited states.
Experimental Setup
The authors evaluated S-Adapt across a diverse range of environments to test its adaptability to different entropy regimes:
- Didactic Environments: Custom tasks including a Maze (low-entropy, testing curiosity), Tetris (high-entropy, testing stability/survival), and Butterflies (a grid world with moving targets).
- MinAtar: A simplified Atari benchmark suite used for efficient RL testing.
- Atari Freeway: A complex, image-based environment used to evaluate the scalability of the method to high-dimensional visual inputs.
Key Insights and Results
Versatility and Robustness
S-Adapt demonstrated the ability to match or outperform single-objective agents across nearly all tasks. Crucially, it was the only agent capable of consistently solving both the Maze (which requires $S_{max}$) and Tetris (which requires $S_{min}$). This suggests that the controllability metric successfully identifies the optimal strategy for a given environment's physics.
Emergent Complexity in Tetris
In Tetris, the surprise-minimizing objective proved to be a superior learning signal compared to the actual extrinsic reward (clearing rows). By attempting to minimize future "surprising" block placements, the agent naturally learned to keep the board clean, effectively mastering the game through the lens of stability.
Avoiding Degeneracy
S-Adapt successfully bypassed the "Dark Room" and "Noisy TV" traps. In the Maze, the agent recognized that minimizing surprise was trivial and uninformative, prompting a switch to maximization. In Tetris, it identified that maximization led to uncontrollable chaos, prompting a switch to minimization to maintain order.
Limitations and Future Directions
While S-Adapt represents a significant advancement in generalist unsupervised agents, several areas for improvement remain:
- Temporal Granularity: Currently, the bandit selects an objective only at the start of an episode. Future iterations could benefit from intra-episode adaptation, allowing an agent to switch from exploration to stabilization dynamically.
- Distribution Modeling: The agent uses relatively simple statistical models (Bernoulli or Gaussian) to estimate state probability. More complex, non-linear environments may require more sophisticated density estimation techniques.
- Computational Cost: Maintaining the state marginal distribution and the MAB loop adds overhead compared to standard reinforcement learning architectures.
Conclusion
S-Adapt provides a unified framework for intrinsic motivation by reframing curiosity and stability as two sides of the same "entropy control" coin. By rewarding deviations from random behavior, the agent achieves a level of environmental adaptation previously requiring manual tuning, marking a step forward for autonomous agent development.
Summary updated