Disentangling Engagement from Utility in Recommendation Systems

40 concepts28 papers1 proposal

In the design of search and recommendation engines, the most accessible signal of success is engagement. When a user clicks a link, likes a post, or watches a video, the system receives an immediate, measurable data point. However, a growing body of research suggests that these behavioral signals are often poor proxies for the actual utility or value a user derives from content.

This "engagement-utility misalignment" creates a fundamental challenge: systems optimized to maximize clicks may inadvertently promote clickbait, prioritize impulsive reactions over long-term satisfaction, and diverge from expert-defined standards of quality.

The Information Gap Between Click and Consumption

The primary reason engagement fails as a proxy for utility is a temporal and informational mismatch. As Joachims et al. observed in their seminal study, a click is a response to a snippet—the title, summary, or thumbnail—rather than the content itself. This allows misleading summaries to generate "false positive" signals.

Wu et al. further disentangle this by modeling news recommendation as two distinct phases: click preference (driven by headlines) and reading satisfaction (driven by content body). A user might click out of curiosity only to find the content unsatisfactory, a phenomenon Noel et al. found so prevalent that even a model with perfect click-prediction accuracy might only achieve 66% accuracy in predicting actual positive sentiment. Liu et al. argue that failing to model these two phases separately prevents systems from identifying whether a click was a genuine success or a result of misleading metadata.

System 1 vs. System 2 Interactions

The divergence between engagement and utility can also be understood through the lens of human psychology. Agarwal et al. propose a framework of System-2 Recommenders, distinguishing between "System-1" impulsive engagement (instinctive clicks) and "System-2" true utility (deliberate satisfaction).

When platforms optimize for immediate metrics like watch time or click-through rates, they often capture these impulsive reactions. Milli et al. suggest treating "value" as an unobservable latent variable that must be anchored by explicit feedback, such as "See Less Often" buttons, rather than just observable clicks. Without this distinction, Immorlica et al. show that engagement-based optimization creates a landscape where quality and clickbait become strategic complements, often reducing overall user welfare.

The Reality Gap in Evaluation

The misalignment is most visible when comparing models trained on user behavior against those evaluated by experts. Althammer et al. found a low statistical correlation between system rankings derived from implicit user feedback and those derived from expert-verified "gold standards."

This "reality gap" persists even in large-scale industrial settings. Hager et al. demonstrated that while Unbiased Learning to Rank (ULTR) techniques can effectively improve click prediction on Baidu's search logs, these gains rarely translate to better performance on expert-annotated relevance. Similarly, Craswell et al. and Li et al. note that models optimized solely on behavioral data often fail to capture the nuanced criteria used by human experts to determine information quality.

Strategic User Behavior

Finally, the assumption that engagement signals are "honest" reflections of taste is being challenged. Cen et al. provide evidence of strategization, where users adapt their behavior—skipping songs they like or over-liking others—to "train" the algorithm. In these cases, high engagement might not signal satisfaction, but rather a user's effort to correct a poorly performing engine.

Conversely, a lack of engagement does not always signal a lack of utility. Chuklin highlights the phenomenon of "good abandonment," where a user finds the information they need directly on the search results page (e.g., in a sidebar or snippet) and leaves without clicking. In this scenario, traditional engagement metrics would incorrectly flag the search as a failure, despite the user being fully satisfied.

Moving Beyond Clicks

To bridge this gap, researchers are moving toward multi-objective frameworks. Kitada demonstrates the utility of predicting clicks and conversions (actual purchases) simultaneously, while Liu et al. argue for an Interactions Proportional to Like (IPL) criterion to ensure items receive engagement that matches their true quality. By disentangling the impulse to click from the experience of value, these approaches aim to create systems that serve a user's long-term interests rather than just their immediate attention.

Key Foundational Work

Accurately interpreting clickthrough data as implicit feedback (2005) — contributed 1 concept elements to this direction.
Evaluating Search Engines by Modeling the Relationship Between Relevance and Clicks (2007) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_engagement_utility_misalignment_0f696d0d,
  title = {engagement-utility misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/engagement-utility-misalignment/rdr_0f696d0dfda3480eb308cbf5f6a09012},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_engagement_utility_misalignment_0f696d0d_v1,
  title = {engagement-utility misalignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/engagement-utility-misalignment/rdr_0f696d0dfda3480eb308cbf5f6a09012/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake