Truthful Aggregation of LLMs with an Application to Online Advertising

Ermis Soumalias, Michael J. Curry, Sven Seuken

Introduction

As Large Language Models (LLMs) transition from conversational assistants to primary interfaces for information retrieval, platforms face a complex challenge: integrating commercial interests without compromising the integrity of the user experience. Traditional online advertising relies on auction mechanisms designed for discrete slots. However, LLM-generated content is fluid and holistic, rendering token-by-token bidding or simple slot-based auctions ineffective.

This paper introduces MOSAIC (Mechanism for Optimally Sampling and Aggregating LLM Outputs with Incentive Compatibility), a novel framework designed to truthfully aggregate advertiser preferences into LLM responses. MOSAIC addresses the "exposure problem"—where advertisers might pay for partial mentions that are ultimately negated by the model later in the sentence—and ensures that the platform maximizes social welfare while maintaining alignment with a neutral, user-centric reference model.

The MOSAIC Mechanism

MOSAIC treats the auction process as a mathematical optimization problem, specifically framing it as an importance-sampling task. The mechanism balances three competing objectives: providing helpful answers to users, allowing advertisers to influence output, and ensuring platform revenue through truthful bidding.

Objective and Allocation

The core objective is to maximize a social welfare function that combines advertiser rewards while penalizing significant deviations from a "reference LLM" (an unbiased model representing the user's best interest). Mathematically, this mirrors the Reinforcement Learning from Human Feedback (RLHF) objective.

The allocation process follows a three-step sampling approach:

  1. Candidate Generation: Instead of generating a single response, the system produces $M$ candidate replies using a "context-aware" LLM. This model is prompted with advertiser descriptions to ensure relevant brands are represented in the pool.
  2. Reward Calculation: The mechanism evaluates the reward each advertiser would assign to each specific candidate.
  3. Probabilistic Sampling: A final response is sampled from the candidates using a distribution that accounts for both the advertiser rewards and the original token probabilities of the reference LLM.

Incentive Compatibility and Payments

To prevent strategic manipulation—where advertisers misreport their value to pay less—MOSAIC employs Rochet payments. This generalization of truthful bidding logic ensures that an advertiser’s dominant strategy is to report their true preferences. A critical innovation in MOSAIC is the inclusion of an advertiser-specific utility offset. This ensures that an advertiser’s profit is directly proportional to their actual contribution to the response's quality, effectively discouraging "free-riding" by irrelevant participants.

Experimental Evaluation

The researchers evaluated MOSAIC across 50 diverse user queries, ranging from musical instruction to skincare routines.

  • Models: Experiments primarily utilized Llama-2-7b-chat-hf, with verification performed on Google’s Flan-T5-Large to ensure results hold across different model architectures.
  • Advertiser Profiles: Each advertiser was represented by a name and a brief service description, which informed the "advertiser LLMs" used for preference modeling and candidate generation.

Key Results

  • Rapid Convergence: The mechanism reaches the theoretically optimal distribution with as few as 20 candidate replies. This efficiency is largely due to the "context-aware" generation, which populates the candidate pool with relevant mentions more effectively than vanilla sampling.
  • Computational Efficiency: The total compute cost is equivalent to approximately five standard LLM queries. Because candidate generation is independent, the process can be parallelized, resulting in no additional perceived latency for the end-user.
  • Strategyproofness: A significant breakthrough is that MOSAIC remains strategyproof even when the sampling process has not fully converged. This makes the mechanism resilient to the computational constraints inherent in real-time inference.
  • API-Only Practicality: MOSAIC does not require access to internal model weights or gradients. It functions entirely through standard API calls, making it applicable to proprietary models like GPT-4.

Conclusion and Future Work

MOSAIC provides a path forward for the monetization of generative AI that respects the utility of all stakeholders. By framing LLM aggregation as a truthful sampling problem, it avoids the pitfalls of traditional auction mechanics in the context of fluid text.

Future research directions include the development of dynamic candidate estimation to further reduce costs and the application of this framework to "reverse auctions." In such a scenario, the mechanism could be used to compensate content creators whose data is summarized in LLM outputs, potentially resolving ongoing tensions between AI platforms and publishers.

Summary updated

Abstract

The next frontier of online advertising is revenue generation from LLM-generated content. We consider a setting where advertisers aim to influence the responses of an LLM to align with their interests, while platforms seek to maximize advertiser value and ensure user satisfaction. The challenge is that advertisers' preferences generally conflict with those of the user, and advertisers may misreport their preferences. To address this, we introduce MOSAIC, an auction mechanism that ensures that truthful reporting is a dominant strategy for advertisers and that aligns the utility of each advertiser with their contribution to social welfare. Importantly, the mechanism operates without LLM fine-tuning or access to model weights and provably converges to the output of the optimally fine-tuned LLM as computational resources increase. Additionally, it can incorporate contextual information about advertisers, which significantly improves social welfare. Through experiments with a publicly available LLM, we show that MOSAIC leads to high advertiser value and platform revenue with low computational overhead. While our motivating application is online advertising, our mechanism can be applied in any setting with monetary transfers, making it a general-purpose solution for truthfully aggregating the preferences of self-interested agents over LLM-generated replies.
BibTeX
@inproceedings{soumalias2025truthful,
  author = {Ermis Soumalias and Michael J. Curry and Sven Seuken},
  title = {{Truthful Aggregation of LLMs with an Application to Online Advertising}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=tD9mebUimw},
}
Figure