TL;DR
This paper introduces RARE, a novel framework for real-time ad retrieval that uses LLM-generated "Commercial Intentions" (CIs) as an intermediate semantic representation. By leveraging a customized LLM trained with commercial knowledge, RARE significantly improves ad retrieval efficiency and effectiveness, leading to substantial gains in online advertising metrics.
What Problem Does This Paper Address?
Traditional ad retrieval systems often rely on multi-stage processes involving keyword extraction and matching, which can lead to missed opportunities and semantic inefficiencies. Existing LLM-based retrieval methods, while promising, often struggle with scalability due to the overhead of generating and mapping numeric or content-based document IDs (DocIDs). This is particularly problematic in large-scale advertising systems that handle millions of ads and queries daily, where real-time performance is critical. The core challenge is to bridge the gap between user queries and relevant ads more efficiently and semantically, while also overcoming the limitations of traditional keyword-based approaches and the scalability issues of current LLM retrieval methods.
Methods
The RARE framework introduces a novel approach centered around "Commercial Intentions" (CIs) – short, semantically rich text snippets generated by a customized LLM. The process involves two main stages:
- Customized LLM: A base LLM is enhanced through two key steps:
- Knowledge Injection: Commercial and advertising domain-specific knowledge is injected into the LLM to improve its understanding of the advertising landscape.
- Format Fine-Tuning: The LLM is fine-tuned to exclusively output CIs, optimizing for decoding efficiency and ensuring a consistent output format.
- RARE Framework:
- Offline Indexing: The customized LLM generates CIs for all ads in the corpus, creating a dynamic index that maps these CIs to their corresponding ads. This mapping is a one-to-many relationship, making it lightweight and scalable.
- Online Retrieval: When a user query arrives, the system first checks an offline cache for high-frequency queries. If not found, the customized LLM generates CIs for the query in real-time. These query CIs are then used to efficiently retrieve relevant ads from the pre-built index.
- Constrained Beam Search: This technique is employed during CI generation to ensure that the outputs are confined to a predefined set of relevant intents, enhancing diversity and relevance.
Datasets
The paper does not explicitly name specific public datasets used for training or evaluation. However, it mentions that:
- Training Data: For knowledge injection, data was derived from advertising systems and synthesized. For format fine-tuning, data was sourced from real-world online logs and processed.
- Evaluation Data: A dataset of 5,000 head queries and 150,000 ads was collected from real-world online scenarios for offline evaluation.
Key Insights and Ideas
- Commercial Intentions as a Semantic Bridge: The core innovation is using LLM-generated text (CIs) as an intermediate representation. This moves beyond simple keyword matching or abstract numeric IDs, allowing for a richer semantic understanding of both user queries and ad content.
- Scalability through One-to-Many Mapping: By mapping multiple ads to a single CI, the framework achieves a lightweight and scalable index, crucial for handling massive ad inventories in real-time.
- Domain-Specific LLM Customization: Injecting commercial knowledge and fine-tuning the LLM's output format are critical for generating relevant and efficient CIs, demonstrating the power of tailored LLMs for specific industry problems.
- Real-time Performance Achieved: The combination of caching, efficient CI generation, and a streamlined indexing process allows RARE to achieve real-time retrieval in a production environment handling millions of daily requests.
Key Results
- Online A/B Testing: RARE demonstrated significant improvements across various online advertising metrics:
- Consumption: +5.04%
- Gross Merchandise Volume (GMV): +6.37%
- Click-Through Rate (CTR): +1.28%
- Shallow Conversions: +5.29%
- Deep Conversions: +24.77%
- Offline Evaluation: RARE outperformed ten competitive baselines across four major categories, showing superior performance in metrics like HR@500 and MAP, while maintaining a high Ad Coverage Rate (ACR) of 95.05%.
- Ablation Studies: Removing knowledge injection (w/o. KI) significantly reduced recall, highlighting the importance of domain-specific knowledge. Removing constrained beam search (w/o. CBS) drastically reduced the number of generated CIs, underscoring its role in diversity.
Limitations and Future Work
- Limitations:
- The current RARE framework primarily focuses on generating CIs and then retrieving ads. The correlation assessment between query and ad is handled by downstream processes.
- While RARE achieves good results, the paper acknowledges that the integration of generative and discriminative capabilities within LLMs could further enhance performance.
- Future Work:
- Integrate correlation assessment directly into LLMs to concurrently evaluate the pertinence of prompts and commercial intents during generation. This would empower the model to evaluate query-ad relevance more holistically.
Summary updated