Apparently Irrational Choice as Optimal Sequential Decision Making

Haiyang Chen, Hyung Jin Chang, Andrew Howes

Introduction

In behavioral economics, human decision-making is frequently characterized as "predictably irrational." We often deviate from the utility-maximizing behavior predicted by classical economic models, falling prey to marketing tactics and context-dependent shifts in preference. However, a recent paper presented at AAAI-21, "Apparently Irrational Choice as Optimal Sequential Decision Making," proposes a shift in perspective. It suggests that these "biases" are not flaws, but rather the hallmarks of a computationally rational system designed to make high-quality decisions under constraints of time, noise, and cognitive cost.

The Problem: Context Effects and "Irrationality"

Psychologists have long documented context effects, where the introduction of a third, seemingly irrelevant option (C) alters the preference between two original options (A and B). Two prominent examples include:

  • The Attraction Effect: A preference for a target option increases when a "decoy"—an option clearly inferior to the target—is introduced.
  • The Compromise Effect: A tendency to choose the "middle" option to avoid extremes.
Illustration of the Attraction Effect: The presence of a 'decoy' cake makes the target cake more appealing compared to an apple.

Standard normative theories view these shifts as irrational, as the intrinsic value of an apple should not change based on the existence of a poor-quality cake. This research seeks to explain why such behavior might be evolutionarily advantageous by framing choice as a problem of computational rationality.

Methodology: Choice as a POMDP

The authors modeled the decision-making process as a Partially Observable Markov Decision Process (POMDP), solved using a Deep Reinforcement Learning agent (ACER algorithm).

The agent is equipped with specific "cognitive actions":

  1. Calculation: Estimating the absolute value of an individual option.
  2. Comparison: Directly comparing two options to determine which is superior.
  3. Choice: Finalizing the decision and terminating the process.

Crucially, every cognitive action incurs a computational cost, and all observations are noisy. The agent's objective is to maximize the expected value of the chosen item while minimizing the total cost of deliberation. The model was trained on "Ecological Distributions"—simulated choice sets reflecting real-world value distributions from landmark behavioral studies (e.g., Wedell 1991, Trueblood 2012).

Key Results: Replicating Human Biases

The "Integrated Agent" successfully replicated the three major context effects—Attraction, Compromise, and Similarity—using a single, fixed policy.

Model results showing the replication of Attraction, Compromise, and Similarity effects through choice probabilities for Target, Competitor, and Decoy options.

Key insights from the model include:

  • Efficiency of Comparison: The agent discovered that direct comparison is often more reward-efficient than absolute calculation, especially for similar items.
  • Decoys as Information Landmarks: The attraction effect emerges because a dominated decoy provides a low-cost reference point, reducing uncertainty about the target option.
  • Emergent Bias: The agent was not programmed to be biased; rather, the optimal policy for a noisy, cost-constrained environment naturally exhibits context sensitivity.
  • Sensitivity to Noise and Cost: As the cost of thinking increases, context effects weaken as the agent stops gathering information earlier. Conversely, higher calculation noise leads to a greater reliance on comparisons, strengthening these effects.

Conclusion

This research suggests that "irrational" human choices are actually the result of a highly efficient sequential decision-making process. By balancing the accuracy of a choice against the computational cost of making it, our brains use clever shortcuts to navigate a complex world. What looks like a bias from a distance is, in fact, an optimal strategy for a system where thinking is never free.

Summary updated

Abstract

In this paper, we propose a normative approach to modeling apparently human irrational decision making (cognitive biases) that makes use of inherently rational computational mechanisms. We view preferential choice tasks as sequential decision making problems and formulate them as Partially Observable Markov Decision Processes (POMDPs). The resulting sequential decision model learns what information to gather about which options, whether to calculate option values or make comparisons between options and when to make a choice. We apply the model to choice problems where context is known to influence human choice, an effect that has been taken as evidence that human cognition is irrational. Our results show that the new model approximates a bounded optimal cognitive policy and makes quantitative predictions that correspond well to evidence about human choice. Furthermore, the model uses context to help infer which option has a maximum expected value while taking into account computational cost and cognitive limits. In addition, it predicts when, and explains why, people stop evidence accumulation and make a decision. We argue that the model provides evidence that apparent human irrationalities are emergent consequences of processes that prefer higher value (rational) policies.
BibTeX
@inproceedings{chen2021apparently,
  author = {Haiyang Chen and Hyung Jin Chang and Andrew Howes},
  title = {{Apparently Irrational Choice as Optimal Sequential Decision Making}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i1.16161},
}
Figure