A mediator can steer self-interested learning agents toward a good outcome without rewriting the game they play. It privately recommends each agent an action and promises a small payment that makes following the advice worthwhile. When the mediator does not know what the agents value, recent work shows that the payments can also act as probes. The mediator adjusts them until agents become indifferent, and at that point the payments reveal the agents' hidden utilities. Every guarantee for this trick assumes the utilities never change. In recommender markets, ad platforms and creator economies, they change constantly, partly because of the mediator's own recommendations. They also live over catalogs far too large to probe one joint action at a time.
We propose a mediator whose payments track utilities as they move. The approach rests on one structural bet: an agent's utility can be written as a slowly drifting weight vector applied to fixed multimodal item features, meaning image and text embeddings of the kind multimodal recommenders already compute. If the bet holds, the mediator only has to follow a few dozen moving weights instead of a table that grows exponentially with the number of agents. The drift then shows up directly in the payment trajectory, which becomes a measuring instrument. It can also tell apart drift the users would have undergone anyway from drift the mediator caused. The bet can be tested on a semi-synthetic creator market built from real drift and real item features.
Core Claim
The setting is a correlated equilibrium with payments (CEP). Each round, a mediator draws a joint action profile from a target distribution, privately tells each agent its part, and pays just enough that obeying is a best response. Learning a Game by Paying the Agents shows that a mediator facing no-regret learners with unknown, fixed utilities can learn those utilities up to strategic equivalence. It does so by running projected gradient descent on payments in a zero-sum meta-game, and it needs $M^{O(1)}/\varepsilon^2$ rounds, where $M$ is the number of joint action profiles. That paper and Steering No-Regret Learners to a Desired Equilibrium, which introduced directness bonuses and sandboxing payments, both assume stationary payoffs. They also treat actions as abstract indices.
We claim that under the factorization
$$u_i^t(a) = \langle \theta_i^t, \phi(a) \rangle + r_i(a),$$
the payment-probing approach can be made to track utilities. Here $u_i^t(a)$ is agent $i$'s utility at round $t$ for joint profile $a$. The map $\phi(a) \in \mathbb{R}^d$ is a fixed feature map built from frozen multimodal item embeddings and the congestion structure of the profile. The vector $\theta_i^t$ is a drifting weight vector, and $r_i$ is a small, time-invariant residual. The tracking version restricts the payment updates to the span of $\phi$ and runs them with discounting or restarts. Its cumulative welfare gap relative to the per-round optimal CEP should then scale with $d$ rather than $M$, and with the drift path length $P_T = \sum_t \|\theta^{t+1} - \theta^t\|$. The expected form comes from the standard dynamic-regret bound for online gradient descent, roughly $\sqrt{T(1+P_T)}$. That bound is not from the file, and the full bound would also have to account for the agents' own learning lag.
The second, empirical claim is that the drift is readable. A mediator that occasionally randomizes its recommendations can split the observed movement in $\theta$ into two parts. One is natural drift, meaning what users would have done under random exposure. The other is drift induced by the mediator's own steering. This split decides whether the payment budget keeps shrinking or starts growing.
The construction differs from three neighbors. Regret Minimization in Stackelberg Games with Side Information uses context vectors, but only for a single leader's commitment against stationary or adversarial follower types, with no payments and no correlated recommendations. Steering No-Regret Agents in MFGs under Model Uncertainty handles unknown rewards with pessimistic estimates and a low-switching mechanism, but it assumes rewards are fixed and states are abstract. Estimating and Penalizing Preference Shift in Recommender Systems measures and penalizes drift caused by the recommender, but it has no agents with strategic incentives and no payments. We reuse its notion of natural preference shift, the drift that would occur under uniformly random recommendations, as the reference point for separating the two kinds of drift.
Research Program
The first component is the market. Following Supply-Side Equilibria in Recommender Systems, we model $N$ creators or sellers who repeatedly choose which item, or which region of the catalog, to supply. That paper shows that unmediated creators can collapse onto a single genre that serves the average user. Here, a creator's payoff is the audience share its item attracts minus a production cost. Audience demand comes from user preferences over item features that drift over time. Creators who pick similar items split their audience, which creates congestion.
The creators run standard no-regret learners: Hedge and regret matching. The mediator's target is the welfare-optimal correlated allocation at the current preferences. In practice, this target spreads creators across niches so that more users are served.
The second component is the representation. Following A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation, we freeze a $k$-nearest-neighbor item graph built from raw visual and text features. That paper reports that freezing beats the dynamically learned graph of Mining Latent Structures for Multimedia Recommendation by 19.07% on average in Recall@20 and NDCG@20, at up to 4× lower training time. We then smooth the features over the graph to obtain $\phi$.
Freezing matters for a specific reason. If both the features and the weights moved, the mediator could not tell genuine preference drift apart from changes in its own representation. With frozen features, all drift is forced into $\theta$. This is our resolution of the tension between the dynamic graph learning of LATTICE and the freezing of FREEDOM: freeze the geometry of the items and let only the preference weights move.
The third component is the mediator algorithm. It extends the zero-sum payment game of Learning a Game by Paying the Agents in four ways.
- Projected updates. Payment updates are projected onto the span of $\phi$, plus a small tabular residual term that absorbs misspecification.
- Forgetting. Gradients are exponentially discounted. A change detector on the payment residuals triggers restarts when drift is abrupt.
- Payments. Directness and sandboxing payments are computed from the current estimate, with a safety margin that grows with the estimated drift rate.
- Exploration. A small fraction of rounds use uniformly random recommendations.
The random rounds serve as the counterfactual "natural drift" arm used in Estimating and Penalizing Preference Shift in Recommender Systems. They let us estimate how much of the observed drift in $\theta$ was induced by steering.
The fourth component is analysis. The plan is to prove tracking bounds for full mixed-strategy feedback in normal-form games, where the base results are cleanest. We then report trajectory feedback empirically, since Steering No-Regret Learners to a Desired Equilibrium already shows that steering under trajectory feedback needs a growing budget even without drift.
We also expect an impossibility result in the style of the adversarial impossibility in Regret Minimization in Stackelberg Games with Side Information. If $P_T$ grows linearly in $T$, no mediator should be able to keep its average budget vanishing. The goal is therefore to characterize the rate as a function of drift, not to escape that limit.
Baselines
The naive reference is no mediation. Creators run their learners on the raw game. We report the welfare they reach and how concentrated their genre choices become, which shows whether a homogenization failure exists in our market at all.
The strongest existing method is the stationary payment-probing mediator of Learning a Game by Paying the Agents, run on the tabular game. It proves that time-averaged value converges to the optimal CEP value at rate $T^{-1/4}$. We found no reported result on Lacuna for it, or for any steering method, on an external dataset, so its empirical numbers will come from our own runs. Beating it means a lower cumulative dynamic welfare gap at an equal or lower cumulative payment. The comparison must hold under drift, and the method must not lose under stationarity, where the baseline's guarantee applies.
The second strong competitor is a sliding-window or restarted version of the same tabular mediator. It has the same drift-handling as our method but no side information. A third is a fixed sandboxing mediator that always overpays enough to make the target dominant under a worst-case utility range. This is the reference cost used by Steering No-Regret Agents in MFGs under Model Uncertainty. It is robust to drift by construction, so it can only be beaten on budget.
The key ablations isolate each ingredient. Replacing $\phi$ with random features of the same dimension tests whether the multimodal content matters, or merely the low dimension. Removing discounting and restarts tests the drift-tracking machinery. Removing the random-recommendation rounds tests the natural-versus-induced split.
All methods see the same environment seeds, the same agent learners and the same horizon. Each gets the same hyperparameter search budget, tuned on held-out time periods.
Why not just…?
Why not just estimate utilities passively with a drift-aware multimodal recommender and solve for the CEP offline? This is the most serious threat. If logged choices plus FREEDOM-style features predict demand well, active probing may add nothing. Our answer is partly structural. No-regret agents' choices reveal only which action they prefer, not by how much. The payments needed to secure obedience depend on utility differences, and payments can identify those differences while passive play cannot. The answer is also partly empirical: we run exactly this pipeline as a baseline, with the same features and a drift-aware sequence model. If it matches our mediator on both welfare and budget, we will report that probing is unnecessary in this regime.
Why not just add a sliding window to the existing tabular method? It handles drift but still pays $M^{O(1)}$ probing cost each time it restarts. With moderate drift, each window may be too short to re-identify anything. The scaling experiment below varies $N$ and the catalog size at fixed drift and shows where windowed tabular learning stops converging within a window. If it never stops converging at realistic sizes, the side-information component is not earning its place.
Why not just pay sandboxing transfers large enough to cover any plausible drift? It works, and that is why it is a baseline. Its cost, though, is linear in $T$ by design. The question is whether tracking recovers vanishing average cost. Under endogenous drift that runs against the target, the answer may be no, and tracking could end up as costly as sandboxing. We treat that outcome as a finding about when steering is affordable, not as a failure to explain away.
Proposed Experiments
The environment draws drift and features from separate real sources.
- Drift trajectories. These come from KuaiRand. Because its videos were randomly exposed, user preference trajectories estimated over time are not confounded by the platform's own recommendation policy. This gives realistic natural drift. The KuaiRand paper reports no baseline accuracy tables, so it supplies data only.
- Item features. These come from the Amazon Baby, Sports and Clothing subsets used by LATTICE and FREEDOM, with 4,096-dimensional visual and 1,024-dimensional text features. KuaiRand users' drifting weight vectors are mapped onto these item features through a learned linear alignment.
- Endogenous drift. We use the 2D preference-circle simulator from Estimating and Penalizing Preference Shift in Recommender Systems. There, exposure moves preferences, an unpenalized RL policy raised engagement to 7.49 versus 5.71 for a myopic policy, and the safety metric fell from 9.69 to 6.33. We couple user drift to which creators the mediator steers toward.
Metrics are the cumulative gap to the per-round optimal CEP welfare, cumulative mediator payment, agent obedience rate, and error in tracking $\theta$. We also report genre concentration among creators.
Four checks decide the proposal.
- Scaling. At fixed drift, the payment needed to reach a given welfare gap should grow with $d$ and stay roughly flat as $M$ grows. The tabular baselines should show the opposite.
- Abrupt shift. Under a planted change in preferences, recovery time should drop with restarts and match the change-detector delay.
- Endogenous drift. We compare the budget trajectory when induced drift points toward versus away from the target. We also check that the random-round estimate of induced drift agrees with the simulator's ground truth. If budgets grow linearly whenever drift points away from the target, that becomes the paper's central negative result.
- Misspecification. We sweep the size of the tabular residual $r_i$ and show where the factorized mediator falls behind the windowed tabular one. This sets out honestly when item features are and are not enough.
References
- Learning a Game by Paying the Agents
- Steering No-Regret Learners to a Desired Equilibrium
- Regret Minimization in Stackelberg Games with Side Information
- Steering No-Regret Agents in MFGs under Model Uncertainty
- Estimating and Penalizing Preference Shift in Recommender Systems
- Supply-Side Equilibria in Recommender Systems
- A Tale of Two Graphs: Freezing and Denoising Graph Structures for Multimodal Recommendation
- Mining Latent Structures for Multimedia Recommendation
- KuaiRand: An Unbiased Sequential Recommendation Dataset with Randomly Exposed Videos