In the traditional Reinforcement Learning (RL) paradigm, agents are driven by a "carrot and stick" approach—extrinsic reward signals defined by a designer. However, biological agents often thrive in environments where rewards are sparse or non-existent. They don't just wander aimlessly; they learn to prefer certain states that ensure their survival and predictability.
A recent paper, "Exploration and preference satisfaction trade-off in reward-free learning," introduces Pepper, a mechanism that allows artificial agents to autonomously learn these internal preferences. By leveraging the principles of Active Inference, the authors demonstrate how agents can naturally navigate the tension between exploring the unknown and settling into preferred routines.
TL;DR
Pepper is a reward-free framework that enables Bayesian agents to learn internal preferences over states and outcomes using conjugate priors. This approach allows agents to autonomously balance exploration and preference satisfaction based on environmental volatility, mimicking biological survival strategies without needing predefined reward functions.
What Problem Does This Paper Address?
Most AI agents require a carefully tuned reward function to learn meaningful behavior. In complex, real-world environments, defining these rewards is notoriously difficult and often leads to "reward hacking" or brittle performance.
Biological systems, by contrast, rely heavily on intrinsic motivation. They seek out states that are "self-evidencing"—states that confirm their internal model of the world. The challenge is creating an artificial agent that can:
- Learn what it "prefers" without being told.
- Adapt those preferences when the environment changes (volatility).
- Decide when to stop exploring and start satisfying those learned preferences.
Methods
The authors build their approach on Deep Active Inference, a framework where agents act to minimize "Expected Free Energy" (EFE). EFE is a powerful objective because it naturally combines two terms:
- Intrinsic Value: The drive to resolve uncertainty (exploration).
- Extrinsic Value: The drive to reach preferred states (preference satisfaction).
The core innovation is Pepper, a two-step preference learning mechanism:
- Interaction: The agent interacts with the environment, storing a history of observations and latent state representations.
- Accumulation: After an episode, the agent updates its internal "prior preferences" using conjugate priors (specifically, Dirichlet distributions). This acts like a form of Hebbian plasticity—the more an agent experiences a state, the more it "prefers" it, provided the environment remains stable.
The agent is instantiated as a Recurrent State-Space Model (RSSM), similar to the popular Dreamer architecture, allowing it to plan in a latent space.
Datasets
The researchers evaluated Pepper in two distinct settings:
- Modified FrozenLake (16x16): A grid-world environment where the standard reward signal was removed. They introduced "sub-goal" tiles and varied the volatility by changing the map layout every $K$ steps.
- 3D TileWorld (MiniWorld): A more complex 3D environment where the agent receives raw pixel observations (32x32 RGB). This tested the scalability of preference learning to high-dimensional visual inputs.
Key Insights and Ideas
- Self-Evidencing as a Survival Strategy: In static environments, agents quickly converge on a "favorite" patch of the world. Because they see the same things repeatedly, their Dirichlet priors become very precise (confident), leading them to stay put and satisfy their preferences.
- Volatility Drives Curiosity: When the environment is volatile (the map changes frequently), the agent can never become fully confident in its preferences. This persistent uncertainty keeps the "intrinsic value" term high, effectively forcing the agent to remain in an exploratory mode.
- The Inverted U-Curve: The authors found a non-linear relationship between volatility and exploration. At moderate volatility, agents explore the most. Interestingly, at 100% volatility (where the map changes every step), agents sometimes shift back to satisfying "bi-modal" preferences—clinging to a few predictable locations because the rest of the world is too chaotic to learn.
- Disregarding the "Noisy TV": Pepper agents showed an ability to disregard purely random noise (like continuous map changes that don't affect latent structure) to focus on meaningful state preferences.
Key Results
- Emergent Trade-offs: The experiments quantified the trade-off using Hausdorff distance (a measure of trajectory diversity). Pepper agents successfully transitioned from satisfying preferences (low distance) to exploring (high distance) as volatility increased.
- Reward-Free Success: Even without a single extrinsic reward signal, agents in the 3D TileWorld developed a strong preference for specific visual features (like grey walls) and learned to navigate toward them.
- Precision Matters: The "confidence" (mathematical precision) of the learned priors was the primary driver of behavior. High precision led to exploitation; low precision led to exploration.
Limitations and Future Work
- Designer Control: The biggest strength of Pepper—its reward-free nature—is also a limitation. Because the agent chooses its own preferences, a human designer has little control over its ultimate goals. This makes it unsuitable for safety-critical tasks like autonomous driving without further supervision.
- Model Sensitivity: The system is highly dependent on the quality of the underlying Bayesian generative model. If the model fails to capture the environment's structure accurately, the learned preferences will be nonsensical.
- Category Space: The authors noted that learning preferences requires a "rich category space." Future work could look into how agents might autonomously discover or expand these categories as they encounter new phenomena.
For those interested in the implementation details, the authors utilized Dreamer V2 as a base and required approximately 1344 GPU hours on RTX 3090s to conduct these experiments.
Summary updated