A Model of Behavioral Manipulation

Daron Acemoğlu, Ali Makhdoumi, Azarakhsh Malekian, Asuman Ozdaglar

TL;DR

This paper provides a formal economic model showing how AI-driven "big data" can be used for behavioral manipulation rather than just better product matching. The authors demonstrate that while AI can help users find better products, it can also harm them by exploiting "glossiness"—temporary attractive features that mask low quality—especially when these deceptive features persist over time or when the variety of available products increases.

What Problem Does This Paper Address?

Economists and tech optimists often argue that more data leads to better product targeting and more informative advertising, ultimately benefiting the consumer. Conversely, critics like Shoshana Zuboff argue that "surveillance capitalism" uses data to modify human behavior for profit.

This paper seeks to bridge that gap by asking: Under what specific conditions does behavioral modification become behavioral manipulation? The authors define manipulation as the "bad" type of modification where a platform uses its superior information to steer users toward low-quality products that appear attractive in the short term, ultimately reducing consumer welfare.

Methods

The authors develop a theoretical behavioral model using a multi-armed bandit framework in continuous time. Key components include:

  • Glossiness ($\alpha$): A temporary state where a low-quality product masks its "bad news" (e.g., a sleek interface hiding a predatory subscription model). Glossiness transitions to a non-glossy state at a rate $\rho$.
  • Learning Dynamics: Users learn about product quality by consuming them. If a product is low-quality and not glossy, "bad news" arrives at rate $\gamma$.
  • Information Asymmetry: The paper compares a Pre-AI environment (where neither the platform nor the user knows the initial glossiness) to a Post-AI environment (where the platform uses big data from other similar users to perfectly estimate a product's initial glossiness).
  • Behavioral Bias: A crucial assumption is that users are Bayesian but not "hyper-rational"—they don't realize the platform has an informational edge and therefore don't adjust their beliefs based on the platform's specific product offers. This is modeled using the Berk-Nash equilibrium concept.

Datasets

This is a theoretical paper and does not utilize empirical datasets. The results are derived through mathematical proofs and equilibrium analysis of the proposed model.

Key Insights and Ideas

  • The Helpfulness vs. Manipulation Trade-off: AI creates two opposing forces. The helpfulness effect occurs when the platform guides users toward high-quality products faster. The manipulation effect occurs when the platform exploits its knowledge of "glossy" low-quality products to extract surplus before the user realizes the quality is poor.
  • The Persistence of Deception ($\rho$): The welfare outcome hinges on how long "glossiness" lasts. If it's short-lived (high $\rho$), the platform can't profit much from deception, so it focuses on helpful matching. If it's long-lived (low $\rho$), the platform finds it more profitable to "trap" the user in a glossy, low-quality experience.
  • The "Double Whammy" of Choice: In standard economics, more choice is always better. Here, as the number of products ($n$) increases, the platform has more opportunities to find "glossy" low-quality items to present to the user. This means that advances in AI that simultaneously increase product variety and predictive power can be twice as harmful to consumers.

Key Results

  • Welfare Gains (High $\rho$): When glossiness is transient, the Post-AI environment increases expected user utility, platform profits, and total utilitarian welfare compared to the Pre-AI status quo.
  • Welfare Losses (Low $\rho$): When glossiness is persistent, the Post-AI environment leads to lower user utility and lower total welfare, even though platform profits increase.
  • Impact of Variety: For a sufficiently small transition rate $\rho$, increasing the number of products $n$ decreases the expected user utility and welfare in the Post-AI environment. The platform's ability to manipulate intensifies with a larger "menu" of potential traps.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Dynamic Learning: The model assumes the platform estimates glossiness perfectly and instantly; real-world platforms must learn this over time.
  • User Sophistication: The model assumes a specific behavioral bias. Future work could explore how varying levels of user "savviness" (understanding the platform's incentives) change the equilibrium.
  • Competition: The current model focuses on a single platform. Introducing competition between platforms might mitigate manipulation if platforms compete on "reputation for helpfulness."
  • Regulation: The paper suggests a need for regulatory frameworks but does not detail specific policy interventions to counter behavioral manipulation.

Summary updated

Abstract

We build a model of online behavioral manipulation driven by AI advances.A platform dynamically offers one of n products to a user who slowly learns product quality.User learning depends on a product's "glossiness,' which captures attributes that make products appear more attractive than they are.AI tools enable platforms to learn glossiness and engage in behavioral manipulation.We establish that AI benefits consumers when glossiness is short-lived.In contrast, when glossiness is long-lived, users suffer because of behavioral manipulation.Finally, as the number of products increases, the platform can intensify behavioral manipulation by presenting more low-quality, glossy products.
BibTeX
@misc{acemoglu2023model,
  author = {Daron Acemoğlu and Ali Makhdoumi and Azarakhsh Malekian and Asuman Ozdaglar},
  title = {{A Model of Behavioral Manipulation}},
  year = {2023},
  doi = {10.2139/ssrn.4638206},
  howpublished = {SSRN},
}

Loading related content...

Figure