The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons

Rachit Dubey, Thomas L. Griffiths, Peter Dayan

Introduction

In psychology, the "hedonic treadmill" describes the human tendency to quickly return to a stable level of happiness despite major life changes. We are often plagued by relative comparisons, valuing our achievements only in relation to our past expectations or the status of others. While these traits are frequently viewed as psychological flaws, this research frames them through the lens of reinforcement learning (RL) to ask: Could habituation and relative comparisons be optimal features designed to help us learn and adapt?

The Reward Design Problem

The authors tackle this question by framing it as a reward design problem. In standard RL, an agent's reward is typically "objective" (e.g., a binary signal for reaching a goal). However, in complex real-world environments, these objective rewards are often sparse or delayed. The researchers investigate whether "subjective" reward functions—those influenced by internal expectations and external comparisons—provide a superior "steering signal" to help an agent master its environment.

Methodology

Utilizing the Optimal Reward Framework, the study distinguishes between the objective reward (the ultimate goal) and the subjective reward (the signal the agent uses to learn). Agents were endowed with a subjective reward function $f$ composed of three weighted components:

  1. Objective ($r_t$): The raw task reward.
  2. Expect ($r_t - Q(s_t, a_t)$): The difference between the reward and the agent's prior expectation, representing habituation.
  3. Compare ($r_t - \rho$): The difference between the reward and a fixed or dynamic aspiration level, representing relative comparison.

The researchers used Q-learning and performed an exhaustive grid search over the weights ($w_1, w_2, w_3$) and the aspiration level ($\rho$) to identify the combinations that maximized the total cumulative objective reward across various scenarios.

Experimental Environments

The study primarily utilized simulated Gridworld environments to test these reward functions. In these 2D spaces, agents navigate to find "food" while avoiding "poison" and "sinkholes."

Legend for Gridworld environment symbols.

Three main variants were tested:

  • Lifetime Learning (Stationary): The food remains in a fixed location.
  • Non-stationary: The food location changes periodically (e.g., every 1250 steps).
  • Sparse: A larger grid where the food is significantly harder to locate.
Gridworld variants: Lifetime learning (stationary), Non-stationary, and Sparse environments.

Additionally, Multi-armed Bandit tasks were used to study the "paradox of choice" and how dynamic aspiration levels affect decision-making when multiple similar options are available.

Key Insights

Dissatisfaction as an Exploration Incentive

Relative comparisons effectively act as a "novelty bonus." By assigning negative subjective value to non-rewarding states the agent has already visited (because they fall below the aspiration level), the agent is pushed to explore unvisited states, leading to faster discovery of goals.

Robustness to Environmental Change

In non-stationary environments, prior expectations prevent the agent from becoming complacent. When a reward disappears due to an environmental shift, the agent receives a sharp negative subjective signal. This prompts the agent to abandon the old location and find the new reward much faster than a standard agent relying solely on objective signals.

The Aversion Trap and Paradox of Choice

The study also identified potential downsides. In sparse environments, if aspiration levels are set too high, agents can develop "aversions" to rewarding states; the transition back to the start after reaching a goal might be perceived as so disappointing that the agent learns to avoid the goal entirely. Similarly, in bandit tasks with many similar options, comparison-based agents remain constantly dissatisfied because no single option significantly outperforms the others, leading to "misery" without any performance gain.

Results and Conclusion

The results were striking: in a stationary 7x7 Gridworld, a "Compare only" agent achieved a mean cumulative objective reward of 2097.02, significantly outperforming the "Objective only" agent's score of 1321.89. Furthermore, subjective agents reached optimal policies much faster, particularly in non-stationary tasks.

Ultimately, the simulations reveal a fundamental trade-off: agents that maximize objective performance often experience moderate to high levels of subjective unhappiness. This suggests that our perpetual dissatisfaction may not be a bug, but a feature—an evolutionary mechanism that keeps us learning and adapting in an ever-changing world.

Summary updated

Abstract

In evaluating our choices, we often suffer from two tragic relativities. First, when our lives change for the better, we rapidly habituate to the higher standard of living. Second, we cannot escape comparing ourselves to various relative standards. Habituation and comparisons can be very disruptive to decision-making and happiness, and till date, it remains a puzzle why they have come to be a part of cognition in the first place. Here, we present computational evidence that suggests that these features might play an important role in promoting adaptive behavior. Using the framework of reinforcement learning, we explore the benefit of employing a reward function that, in addition to the reward provided by the underlying task, also depends on prior expectations and relative comparisons. We find that while agents equipped with this reward function are less happy, they learn faster and significantly outperform standard reward-based agents in a wide range of environments. Specifically, we find that relative comparisons speed up learning by providing an exploration incentive to the agents, and prior expectations serve as a useful aid to comparisons, especially in sparsely-rewarded and non-stationary environments. Our simulations also reveal potential drawbacks of this reward function and show that agents perform sub-optimally when comparisons are left unchecked and when there are too many similar options. Together, our results help explain why we are prone to becoming trapped in a cycle of never-ending wants and desires, and may shed light on psychopathologies such as depression, materialism, and overconsumption.
BibTeX
@article{dubey2022pursuit,
  author = {Rachit Dubey and Thomas L. Griffiths and Peter Dayan},
  title = {{The pursuit of happiness: A reinforcement learning perspective on habituation and comparisons}},
  journal = {PLOS Computational Biology},
  year = {2022},
  doi = {10.1371/journal.pcbi.1010316},
}
Figure