Introduction
In psychology, the "hedonic treadmill" describes the human tendency to quickly return to a stable level of happiness despite major life changes. We are often plagued by relative comparisons, valuing our achievements only in relation to our past expectations or the status of others. While these traits are frequently viewed as psychological flaws, this research frames them through the lens of reinforcement learning (RL) to ask: Could habituation and relative comparisons be optimal features designed to help us learn and adapt?
The Reward Design Problem
The authors tackle this question by framing it as a reward design problem. In standard RL, an agent's reward is typically "objective" (e.g., a binary signal for reaching a goal). However, in complex real-world environments, these objective rewards are often sparse or delayed. The researchers investigate whether "subjective" reward functions—those influenced by internal expectations and external comparisons—provide a superior "steering signal" to help an agent master its environment.
Methodology
Utilizing the Optimal Reward Framework, the study distinguishes between the objective reward (the ultimate goal) and the subjective reward (the signal the agent uses to learn). Agents were endowed with a subjective reward function $f$ composed of three weighted components:
- Objective ($r_t$): The raw task reward.
- Expect ($r_t - Q(s_t, a_t)$): The difference between the reward and the agent's prior expectation, representing habituation.
- Compare ($r_t - \rho$): The difference between the reward and a fixed or dynamic aspiration level, representing relative comparison.
The researchers used Q-learning and performed an exhaustive grid search over the weights ($w_1, w_2, w_3$) and the aspiration level ($\rho$) to identify the combinations that maximized the total cumulative objective reward across various scenarios.
Experimental Environments
The study primarily utilized simulated Gridworld environments to test these reward functions. In these 2D spaces, agents navigate to find "food" while avoiding "poison" and "sinkholes."
Three main variants were tested:
- Lifetime Learning (Stationary): The food remains in a fixed location.
- Non-stationary: The food location changes periodically (e.g., every 1250 steps).
- Sparse: A larger grid where the food is significantly harder to locate.
Additionally, Multi-armed Bandit tasks were used to study the "paradox of choice" and how dynamic aspiration levels affect decision-making when multiple similar options are available.
Key Insights
Dissatisfaction as an Exploration Incentive
Relative comparisons effectively act as a "novelty bonus." By assigning negative subjective value to non-rewarding states the agent has already visited (because they fall below the aspiration level), the agent is pushed to explore unvisited states, leading to faster discovery of goals.
Robustness to Environmental Change
In non-stationary environments, prior expectations prevent the agent from becoming complacent. When a reward disappears due to an environmental shift, the agent receives a sharp negative subjective signal. This prompts the agent to abandon the old location and find the new reward much faster than a standard agent relying solely on objective signals.
The Aversion Trap and Paradox of Choice
The study also identified potential downsides. In sparse environments, if aspiration levels are set too high, agents can develop "aversions" to rewarding states; the transition back to the start after reaching a goal might be perceived as so disappointing that the agent learns to avoid the goal entirely. Similarly, in bandit tasks with many similar options, comparison-based agents remain constantly dissatisfied because no single option significantly outperforms the others, leading to "misery" without any performance gain.
Results and Conclusion
The results were striking: in a stationary 7x7 Gridworld, a "Compare only" agent achieved a mean cumulative objective reward of 2097.02, significantly outperforming the "Objective only" agent's score of 1321.89. Furthermore, subjective agents reached optimal policies much faster, particularly in non-stationary tasks.
Ultimately, the simulations reveal a fundamental trade-off: agents that maximize objective performance often experience moderate to high levels of subjective unhappiness. This suggests that our perpetual dissatisfaction may not be a bug, but a feature—an evolutionary mechanism that keeps us learning and adapting in an ever-changing world.
Summary updated