Incentivizing Exploration for Myopic Agents

This direction is split into other directions: Bayesian Incentive Compatibility. Its last version stays readable and citable here.

Version 1 of 128 papers34 conceptsfrozen 2026-10-04

In a standard multi-armed bandit problem, a single agent balances exploration (trying new things) and exploitation (sticking to what works) to maximize their own long-term reward. But in modern recommendation systems—like Yelp, Netflix, or Waze—this trade-off is split between two parties with conflicting goals.

The platform (the "planner") wants to explore new restaurants or routes to improve the system for everyone. The users (the "agents"), however, are myopic: they only care about their own experience right now. As Mark Sellke notes in Incentivizing Exploration with Linear Contexts and Combinatorial Actions, if a user suspects they are being used as a "guinea pig" to test a risky new option, they will simply ignore the recommendation and stick to what they know.

This research direction explores how to design mechanisms that convince self-interested users to explore for the greater good without using direct financial bribes.

The Information Gap as an Incentive

The primary tool a platform has to influence behavior is information asymmetry. The platform sees the outcomes of thousands of previous users, while an individual user only sees their own history or a limited set of reviews.

Nicole Immorlica, Aleksandrs Slivkins, and their colleagues demonstrate in Incentivizing Exploration with Selective Data Disclosure that a platform can prevent "herding" on suboptimal choices by carefully controlling what data it reveals. By partitioning early users into independent "focus groups" and only showing their results to later users, the platform can ensure that exploration happens naturally.

The intuition is simple: if the platform recommends an item, and the user knows the platform is "Bayesian incentive compatible," the user can infer that the recommendation is likely their best move, even if it involves some exploration. The platform effectively "bundles" the risk of exploration with the value of its superior knowledge.

The Fiduciary Guarantee

A more stringent approach is the fiduciary framework, pioneered by Gal Bahar, Moshe Tennenholtz, and Kevin Leyton-Brown. In Fiduciary Bandits, they argue that a recommendation system should act as a "fiduciary" for the user, promising to never recommend an action that it knows is worse than the user's best independent option.

This creates a "do no harm" constraint. In Efficient Individually Rational Recommender System under Stochastic Order, the authors show that even with this constraint, it is possible to achieve optimal long-term learning. The system explores only when the expected reward of an unknown item is high enough that a rational user would be willing to take the gamble, or when the system can "subsidize" the risk using the trust it has built through previous successful recommendations.

Geometry and Context

In the real world, recommendations aren't just about "Item A vs. Item B." They depend on context—your location, the time of day, or your past preferences. Benjamin Schiffer and Mark Sellke explore this in Geometry Meets Incentives.

When users have "linear contexts" (e.g., they value different features of a product differently), the platform must ensure that the exploratory recommendation is "individually rational" for that specific user's context. This requires a deep understanding of the geometry of the action space. If the platform knows that a user's preferences lie in a certain direction, it can only suggest exploratory items that are "close enough" to the user's perceived optimal choice to maintain trust.

Free Exploration through Human Diversity

While myopic behavior is often seen as a hurdle, it can also be a blessing. Carlos Riquelme and Sven Schmit point out in Human Interaction with Recommendation Systems: On Bias and Exploration that human preferences are naturally diverse.

Because different people like different things, a "suboptimal" item for one person might be the "best" item for another. This inherent diversity provides a form of free exploration. A smart system doesn't need to force anyone to be a guinea pig; it just needs to wait for a user whose natural preferences lead them to try the unknown item. The challenge then shifts from forcing exploration to learning from the diverse choices users make on their own.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_incentivizing_exploration_for_myopic_age_8d3ba84c,
  title = {Incentivizing Exploration for Myopic Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incentivizing-exploration-for-myopic-agents/rdr_8d3ba84ce2644e4891eb07162171decf},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_incentivizing_exploration_for_myopic_age_8d3ba84c_v1,
  title = {Incentivizing Exploration for Myopic Agents},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/incentivizing-exploration-for-myopic-agents/rdr_8d3ba84ce2644e4891eb07162171decf/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}