TL;DR
This paper introduces a novel auction mechanism to aggregate preferences from multiple self-interested agents for LLM-generated content, particularly for online advertising. It ensures truthful reporting and converges to an optimal LLM output without requiring model fine-tuning, while also boosting advertiser value and platform revenue.
What Problem Does This Paper Address?
Large Language Models (LLMs) are becoming ubiquitous, but ensuring their outputs align with diverse and potentially conflicting user preferences is a significant challenge. In applications like online advertising, multiple advertisers want to influence LLM responses to promote their interests. Traditional methods of fine-tuning LLMs for each specific preference set are impractical due to the dynamic nature of participants and the computational cost. This paper addresses the problem of how to aggregate these self-interested preferences in a truthful and efficient manner, especially in scenarios involving monetary transactions.
Methods
The authors propose an auction mechanism that operates without needing access to LLM weights or extensive fine-tuning. Instead, it relies on "API access" to LLMs and post-processing of generated outputs. The core of the mechanism involves:
- Candidate Sequence Generation: A context-aware LLM (informed by advertiser descriptions) generates a set of candidate replies to a user query. This is a key innovation, as it bridges the gap between the reference LLM's general usefulness and advertisers' specific interests.
- Reward Aggregation: Advertisers report their rewards for these candidate sequences. The mechanism calculates an aggregate reward.
- Allocation Rule: A re-weighting of candidate sequences based on their rewards and divergence from a reference LLM's output. This rule is designed to provably converge to the optimal LLM distribution as more candidate sequences are generated.
- Payment Rule: A carefully designed payment rule ensures strategyproofness (truthful reporting is optimal) and individual rationality (agents are incentivized to participate). A novel "agent-specific utility offset" is introduced to align an agent's utility with their contribution to social welfare, preventing "reverse market unraveling."
Datasets
The paper does not introduce new datasets. Instead, it uses synthetic instances for its experiments, consisting of user queries and advertiser profiles. The LLMs used are Llama-2-7b-chat-hf as the reference LLM, and advertiser LLMs are created by adding advertising instructions to this reference LLM.
Key Insights and Ideas
- Context-Aware Generation is Crucial: Using a context-aware LLM to generate candidate sequences, rather than just the reference LLM, significantly accelerates convergence to the optimal distribution and dramatically boosts advertiser rewards. This is because it can directly incorporate advertiser-specific information.
- Truthful Aggregation Without Fine-Tuning: The auction mechanism achieves truthful aggregation and optimal LLM output convergence without the need for computationally expensive LLM fine-tuning for each query or agent.
- "Almost Individually Rational" is Sufficient: While strict individual rationality (guaranteed positive utility from participation) is impossible in this setting while maintaining strategyproofness and convergence, the proposed mechanism achieves an "almost IR" property, ensuring positive utility in most practical scenarios.
- Utility Alignment for Market Viability: The "agent-specific utility offset" is a critical innovation. It ensures that an agent's utility gain is proportional to their contribution to social welfare, which is vital for the long-term success and stability of such a market.
Key Results
- The context-aware mechanism converges to the optimal distribution much faster than the baseline (reference LLM-based) approach. For instance, with only 4 candidate sequences, the context-aware mechanism achieved higher log probability than the baseline with 20 sequences.
- The context-aware mechanism significantly increases advertiser reward gain, with benefits scaling rapidly with the number of generated sequences.
- The proposed payment rule, especially with the agent-specific offset, generates substantial revenue for the auctioneer while still providing positive utility gains for advertisers.
- Experiments show a strong, near-linear correlation between an advertiser's reward gain and their utility gain when the offset is used, indicating a more equitable mechanism.
Limitations and Future Work
- Computational Cost: While the mechanism avoids LLM fine-tuning, it still involves generating multiple candidate sequences, which has computational costs. The paper notes that convergence speed is affected by the number of candidate sequences.
- "Almost IR": The paper acknowledges that strict individual rationality is unattainable while preserving other desirable properties. The "almost IR" property is a theoretical compromise.
- Future Work: The paper suggests that the mechanism can be applied to any setting with monetary transfers where preference aggregation over LLM behavior is needed. Further experimental evaluation of the payment rule and offset is discussed in the appendices.
Summary updated