Introduction
Understanding user intent behind search queries is a fundamental challenge in information retrieval. Search queries are often short, ungrammatical, and highly ambiguous. For instance, a query like "apple" could refer to the consumer electronics giant, the fruit, or a specific local business. While Large Language Models (LLMs) possess vast world knowledge, they frequently struggle with these "naked" queries because they lack the necessary context to disambiguate the user's goal.
To address this, researchers have turned to Retrieval Augmentation (RA), where search results (such as page titles and URLs) are fed back into the model as additional context. However, RA introduces significant computational overhead, making it impractical for real-time production environments. This paper introduces QUILL (Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation), a framework designed to capture the benefits of retrieval augmentation within a tiny, production-ready student model without increasing inference latency.
The Challenge: Context vs. Latency
Retrieval Augmentation significantly improves model performance by providing the "missing" context. However, it faces two primary hurdles in a production setting:
- Latency: Including the top 10 search result titles and URLs can increase the input sequence length by an order of magnitude. Due to the quadratic complexity of the Transformer architecture, this makes the model far too slow for the millisecond requirements of modern search engines.
- The Distillation Gap: Standard knowledge distillation attempts to transfer knowledge from a large "teacher" to a small "student." When the teacher uses RA (long input) and the student does not (short input), the performance gap is often too wide for the student to bridge effectively in a single step.
The QUILL Framework
QUILL overcomes these hurdles through a novel three-stage distillation process involving three distinct model roles:
1. The Professor (RA-LLM)
The process begins with a massive LLM (e.g., mT5) that acts as the "Professor." This model has access to the query plus the titles and URLs of the top 10 search results. This configuration serves as the gold standard for accuracy but is computationally prohibitive for scale.
2. The Teacher (Non-RA LLM)
The "Teacher" is an intermediate-sized LLM that only sees the query (potentially with simple keyword expansions). In the first stage of distillation, the Professor trains the Teacher on a moderately sized dataset (2–4 million examples). The Teacher’s goal is to learn to mimic the deep contextual understanding the Professor gained from the search results, effectively internalizing that external knowledge.
3. The Student
The final "Student" is a tiny, 4-layer Transformer optimized for speed. Because the Teacher is significantly faster than the Professor, it can be used to label a massive dataset of over 100 million examples. This high-volume dataset is then used to train the Student. By using the Teacher as a bridge, the Student benefits from the Professor's retrieval-augmented insights without ever needing to perform retrieval at runtime.
Experimental Results
The researchers evaluated QUILL on two major benchmarks: EComm (a Google e-commerce dataset with 128M unlabeled queries) and Orcas-I (a Bing-based dataset with five intent classes).
Key Findings
- The Power of RA: The study found that a "Base" sized model (220M parameters) equipped with Retrieval Augmentation could outperform an "XXL" model (11B+ parameters) that lacked access to search results. This highlights that context is often more valuable than raw parameter count.
- URLs as High-Signal Context: Interestingly, the authors discovered that URLs were often more helpful than page titles. While titles can be verbose or misleading, URLs provide a structured, consistent signal that clearly indicates the nature of the destination site.
- Performance Gains: On the EComm dataset, the QUILL framework improved the Area Under the ROC Curve (AUC-ROC) by 4.3% and Precision-Recall (AUC-PR) by 4.6%. The final QUILL Student outperformed a standard baseline student by approximately 3 percentage points across both metrics.
Conclusion
QUILL demonstrates that the "world knowledge" contained within massive, retrieval-augmented LLMs can be effectively distilled into tiny models suitable for production. By employing a multi-stage distillation strategy, the framework allows the final model to maintain the speed of a 4-layer Transformer while achieving the accuracy of a much larger, context-aware system. This approach paves the way for more intelligent, context-sensitive search experiences that do not compromise on performance.
Summary updated