TL;DR
This paper proposes that human-like "open-ended" intelligence arises from subjective functions—internalized objectives that are defined by the agent's own state rather than external rewards. By maximizing Expected Prediction Error (EPE), agents are naturally driven to seek "positive surprises," leading them to master new skills and then move on to more challenging goals once the old ones are "played out."
What Problem Does This Paper Address?
In standard Reinforcement Learning (RL), rewards are typically exogenous: they are hard-coded by a programmer or the environment (e.g., "get points for winning a game"). However, humans are adept at synthesizing new goals on the fly—think of a child inventing a game or a hobbyist pursuing a niche world record.
The author, Samuel Gershman, argues that the quest for a single "ultimate" objective function (like reproductive fitness) is insufficient for explaining intelligent behavior. Instead, we need to understand the principles that allow an agent to choose its own objectives. The core challenge is defining a "higher-order" principle that disciplines this choice without simply falling back into the trap of another fixed, external objective.
Methods
The paper introduces a formal framework based on Markov Decision Processes (MDPs) but replaces the fixed reward function with a subjective function.
- Expected Prediction Error (EPE): The agent's utility is defined as the difference between the true expected value ($V$) and the agent's current estimate of that value ($\hat{V}$). Mathematically: $U = V - \hat{V}$.
- The Two-Loop Architecture:
- Inner Loop: For a fixed goal, the agent optimizes its policy to maximize EPE. This is equivalent to maximizing value while the value estimate is "frozen," essentially chasing the "optimism gap."
- Outer Loop: The agent selects new goals ($g^*$) that it expects will yield the highest EPE.
- Hybrid Model: To account for the fact that we still value rewards even after we've mastered a task, the paper proposes a weighted combination of standard value and EPE.
The paper is primarily theoretical and conceptual, drawing extensive parallels between this mathematical formulation and observed phenomena in psychology, neuroscience, and existing machine learning algorithms.
Datasets
Not discussed in the paper. This is a theoretical and synthesis paper. It does not introduce new datasets or perform original computational experiments. Instead, it cites a wide array of empirical findings from existing literature in psychology (e.g., studies on lottery winners, children's play, and pigeon behavior) and neuroscience (e.g., dopamine neuron activity).
Key Insights and Ideas
- Positive Surprise as Utility: The EPE model suggests that agents are attracted to "positive surprise." Once a goal is reached and the value estimate becomes perfect, the EPE drops to zero. This explains why we get bored with mastered tasks—the goal is "played out."
- Conditional Rationality: The paper highlights that humans are "efficiently irrational." We might pick an arbitrary, "useless" goal (like cleaning a toy pig with pretend toothpaste or breaking a window-cleaning record), but once that goal is set, we pursue it with high efficiency.
- Hedonic Adaptation: The model provides a formal account of why lottery winners aren't permanently happier: as their expectations ($\hat{V}$) rise to meet their new reality ($V$), the prediction error (and thus subjective well-being) returns to baseline.
- Information Seeking vs. Avoidance: The EPE framework predicts that agents will seek information when they are pessimistic (to reduce the gap) but might avoid information when they are over-optimistic (to preserve the "positive surprise" of their current state).
- Connection to GAE: The author points out a fascinating link to Generalized Advantage Estimation (GAE) in machine learning. Maximizing EPE is mathematically equivalent to using a GAE estimator where the weighting parameter $\lambda = 1$.
Key Results
As a theoretical work, the "results" are the successful mappings of the EPE model to human behavior:
- Preference for Increasing Reward: Humans prefer sequences of outcomes that improve over time, even if the total reward is lower. EPE explains this because increasing rewards generate a steady stream of positive prediction errors.
- Exploratory Play: The model explains why children (and adults) often choose "high-cost" or "inefficient" paths during play. They aren't being inefficient; they are efficiently pursuing a subjective goal that maximizes their internal EPE.
- Pathological Behaviors: The paper suggests that addiction might be a form of "hijacked" goal selection, where the agent remains highly efficient at pursuing a goal that has become endogenously dominant, even if it is exogenously destructive.
Limitations and Future Work
- Practical Implementation: The paper notes that EPE is not yet a "fully worked out practical algorithm." Implementing this in high-dimensional sensory environments remains a challenge, particularly regarding how to handle noisy or unpredictable regions of state space (the "noisy TV" problem).
- Theory Adequacy: It remains an open question whether EPE alone is a sufficient theory for all human goal selection.
- Computational Complexity: The "outer loop" of goal selection requires predicting future EPE, which may be computationally expensive for complex agents.
- Surrogate Policies: In practice, an agent doesn't know the optimal policy for a new goal before trying it, requiring the use of "surrogate policies" during the goal-selection phase.
Summary updated