In a standard multi-armed bandit problem, an algorithm balances exploration (trying new things) and exploitation (choosing the best known option) to maximize total rewards. However, in real-world systems like Netflix, Yelp, or clinical trials, the "arms" are pulled by self-interested agents—users or patients—who only care about their own immediate outcome.
If a navigation app like Waze suggests a side street just to see if it’s congested, and you end up stuck in traffic, you have been "sacrificed" for the sake of the model’s future accuracy. If users suspect this, they will ignore the app's advice and stick to the highway. This creates a fundamental tension: how can a platform gather the data it needs to improve if it cannot force users to take risks?
Information as a Non-Monetary Subsidy
The core insight across this research is that a central platform can use its information advantage to "subsidize" exploration. While a single user only knows their own past experiences, the platform has a global view of every user's history.
As Aleksandrs Slivkins and Nicole Immorlica describe in Exploration and Persuasion, the platform can act as a "mediator" that strategically filters information. By blending exploratory recommendations (risky) with high-value exploitation recommendations (safe), the platform ensures that, on average, following the recommendation is more rational for the user than choosing an option on their own.
This mechanism, often called Bayesian Incentive Compatibility (BIC), works because the user knows the platform has more data. If the platform says "Try Restaurant X," the user calculates the expected value of X based on the platform's track record. As long as the probability of being assigned a "dud" for exploration is low enough, the user's best move is to trust the system.
The Compounding Asset of Knowledge
How does a system get enough information to start this process? Mark Sellke and Aleksandrs Slivkins characterize information as a compounding asset. A small amount of initial data—perhaps gathered from "altruistic" users or through small initial errors—provides the leverage needed to convince future agents to explore.
In Incentivizing Exploration with Linear Contexts and Combinatorial Actions, Mark Sellke shows that this often requires a two-phase design:
- Initial Data Collection: A phase where the platform gathers enough "knowledge surplus" to be credible.
- Main Loop: A phase where that surplus is used to "pay" for further exploration.
This "knowledge surplus" allows the system to offer recommendations that are personally beneficial to the user while simultaneously gathering data to improve future performance, as explored by Gal Bahar and Moshe Tennenholtz in Recommendation Systems and Self Motivated Users.
Fiduciary Constraints and the Do No Harm Principle
While BIC ensures users are better off on average, it still allows for "sacrificing" some users for the greater good. A more stringent framework, introduced by Gal Bahar, Omer Ben-Porat, Kevin Leyton-Brown, and Moshe Tennenholtz, is the Fiduciary Bandit.
In a fiduciary setting, the system is constrained to never suggest an action that it knows has a lower expected reward than the user's best independent option. This is a "do no harm" guarantee. The algorithm FEE (Fiduciary Explore & Exploit) achieves optimal long-term social welfare while strictly honoring this individual guarantee. This is particularly relevant in high-stakes environments like medical trials, where Aleksandrs Slivkins and Yingkai Li demonstrate that information asymmetry can incentivize participation by slightly skewing treatment probabilities toward better-performing options.
Beyond Simple Choices: Traffic and RL
The principle of incentivized exploration extends to complex, stateful environments. In Exploration and Incentives in Reinforcement Learning, Max Simchowitz and Aleksandrs Slivkins introduce Hidden Hallucination, an algorithm that guides agents through a Markov Decision Process (MDP) by strategically controlling the history of observations shared with new agents.
Similarly, in traffic routing, Emily Meigs, Daron Acemoğlu, and colleagues show that optimal information provision can manage congestion. By promising early "explorers" (who take a risky route) exclusive access to future high-value information, a planner can reward risk-taking without using money.
The Risks of Informational Power
While these mechanisms can improve social welfare, they also open the door to behavioral manipulation. Daron Acemoğlu and others warn that platforms can leverage their informational advantage to steer users toward products that benefit the platform (e.g., high-margin items with "glossy" temporary features) rather than the user. The same asymmetry that enables beneficial exploration can be weaponized if the platform's incentives are not aligned with the users'.