Exploration and preference satisfaction trade-off in reward-free learning

Noor Sajid, Panagiotis Tigas, Alexey Zakharov, Zafeirios Fountas, Karl J. Friston

In the traditional Reinforcement Learning (RL) paradigm, agents are driven by a "carrot and stick" approach—extrinsic reward signals defined by a designer. However, biological agents often thrive in environments where rewards are sparse or non-existent. They don't just wander aimlessly; they learn to prefer certain states that ensure their survival and predictability.

A recent paper, "Exploration and preference satisfaction trade-off in reward-free learning," introduces Pepper, a mechanism that allows artificial agents to autonomously learn these internal preferences. By leveraging the principles of Active Inference, the authors demonstrate how agents can naturally navigate the tension between exploring the unknown and settling into preferred routines.

TL;DR

Pepper is a reward-free framework that enables Bayesian agents to learn internal preferences over states and outcomes using conjugate priors. This approach allows agents to autonomously balance exploration and preference satisfaction based on environmental volatility, mimicking biological survival strategies without needing predefined reward functions.

What Problem Does This Paper Address?

Most AI agents require a carefully tuned reward function to learn meaningful behavior. In complex, real-world environments, defining these rewards is notoriously difficult and often leads to "reward hacking" or brittle performance.

Biological systems, by contrast, rely heavily on intrinsic motivation. They seek out states that are "self-evidencing"—states that confirm their internal model of the world. The challenge is creating an artificial agent that can:

  1. Learn what it "prefers" without being told.
  2. Adapt those preferences when the environment changes (volatility).
  3. Decide when to stop exploring and start satisfying those learned preferences.

Methods

The authors build their approach on Deep Active Inference, a framework where agents act to minimize "Expected Free Energy" (EFE). EFE is a powerful objective because it naturally combines two terms:

  • Intrinsic Value: The drive to resolve uncertainty (exploration).
  • Extrinsic Value: The drive to reach preferred states (preference satisfaction).

The core innovation is Pepper, a two-step preference learning mechanism:

  1. Interaction: The agent interacts with the environment, storing a history of observations and latent state representations.
  2. Accumulation: After an episode, the agent updates its internal "prior preferences" using conjugate priors (specifically, Dirichlet distributions). This acts like a form of Hebbian plasticity—the more an agent experiences a state, the more it "prefers" it, provided the environment remains stable.
Figure 1: Model architecture and 2-step training procedure. Circles and squares denote random and deterministic variables respectively. Coloured lines denote connections where learning is employed. Shaded circles represent outcomes that have already been observed by the agent. The first figure shows the generative model used during learning. The second panel is for Pepper (the preference learning phase) – comprising two steps in each episode: 1) interaction with the environment, & 2) accumulation of preferences once interaction ends. There is a bi-directional flow between the 2 steps: step 1 influences preference learning and step 2 in turn influences environment interaction in the next episode.

The agent is instantiated as a Recurrent State-Space Model (RSSM), similar to the popular Dreamer architecture, allowing it to plan in a latent space.

Datasets

The researchers evaluated Pepper in two distinct settings:

  • Modified FrozenLake (16x16): A grid-world environment where the standard reward signal was removed. They introduced "sub-goal" tiles and varied the volatility by changing the map layout every $K$ steps.
  • 3D TileWorld (MiniWorld): A more complex 3D environment where the agent receives raw pixel observations (32x32 RGB). This tested the scalability of preference learning to high-dimensional visual inputs.

Key Insights and Ideas

  • Self-Evidencing as a Survival Strategy: In static environments, agents quickly converge on a "favorite" patch of the world. Because they see the same things repeatedly, their Dirichlet priors become very precise (confident), leading them to stay put and satisfy their preferences.
  • Volatility Drives Curiosity: When the environment is volatile (the map changes frequently), the agent can never become fully confident in its preferences. This persistent uncertainty keeps the "intrinsic value" term high, effectively forcing the agent to remain in an exploratory mode.
  • The Inverted U-Curve: The authors found a non-linear relationship between volatility and exploration. At moderate volatility, agents explore the most. Interestingly, at 100% volatility (where the map changes every step), agents sometimes shift back to satisfying "bi-modal" preferences—clinging to a few predictable locations because the rest of the world is too chaotic to learn.
  • Disregarding the "Noisy TV": Pepper agents showed an ability to disregard purely random noise (like continuous map changes that don't affect latent structure) to focus on meaningful state preferences.
Visualisation of posterior latent states. In static settings (red), agents cluster in specific preferred areas. In volatile settings (blue), the states are more dispersed, indicating broader exploration.

Key Results

  • Emergent Trade-offs: The experiments quantified the trade-off using Hausdorff distance (a measure of trajectory diversity). Pepper agents successfully transitioned from satisfying preferences (low distance) to exploring (high distance) as volatility increased.
  • Reward-Free Success: Even without a single extrinsic reward signal, agents in the 3D TileWorld developed a strong preference for specific visual features (like grey walls) and learned to navigate toward them.
  • Precision Matters: The "confidence" (mathematical precision) of the learned priors was the primary driver of behavior. High precision led to exploitation; low precision led to exploration.

Limitations and Future Work

  • Designer Control: The biggest strength of Pepper—its reward-free nature—is also a limitation. Because the agent chooses its own preferences, a human designer has little control over its ultimate goals. This makes it unsuitable for safety-critical tasks like autonomous driving without further supervision.
  • Model Sensitivity: The system is highly dependent on the quality of the underlying Bayesian generative model. If the model fails to capture the environment's structure accurately, the learned preferences will be nonsensical.
  • Category Space: The authors noted that learning preferences requires a "rich category space." Future work could look into how agents might autonomously discover or expand these categories as they encounter new phenomena.

For those interested in the implementation details, the authors utilized Dreamer V2 as a base and required approximately 1344 GPU hours on RTX 3090s to conduct these experiments.

Summary updated

Abstract

Biological agents have meaningful interactions with their environment despite the absence of immediate reward signals. In such instances, the agent can learn preferred modes of behaviour that lead to predictable states -- necessary for survival. In this paper, we pursue the notion that this learnt behaviour can be a consequence of reward-free preference learning that ensures an appropriate trade-off between exploration and preference satisfaction. For this, we introduce a model-based Bayesian agent equipped with a preference learning mechanism (pepper) using conjugate priors. These conjugate priors are used to augment the expected free energy planner for learning preferences over states (or outcomes) across time. Importantly, our approach enables the agent to learn preferences that encourage adaptive behaviour at test time. We illustrate this in the OpenAI Gym FrozenLake and the 3D mini-world environments -- with and without volatility. Given a constant environment, these agents learn confident (i.e., precise) preferences and act to satisfy them. Conversely, in a volatile setting, perpetual preference uncertainty maintains exploratory behaviour. Our experiments suggest that learnable (reward-free) preferences entail a trade-off between exploration and preference satisfaction. Pepper offers a straightforward framework suitable for designing adaptive agents when reward functions cannot be predefined as in real environments.
BibTeX
@misc{sajid2021exploration,
  author = {Noor Sajid and Panagiotis Tigas and Alexey Zakharov and Zafeirios Fountas and Karl J. Friston},
  title = {{Exploration and preference satisfaction trade-off in reward-free learning}},
  year = {2021},
  eprint = {2106.04316},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.04316},
}
Figure