TL;DR
This paper introduces a "Gatekeeper" model that acts as a pre-filter for expensive Large Language Models (LLMs) used in text prediction. By intelligently deciding which user inputs are likely to yield good predictions, the Gatekeeper significantly reduces LLM inference costs and improves user experience by avoiding irrelevant suggestions.
What Problem Does This Paper Address?
Modern text prediction (TP) systems often rely on Large Language Models (LLMs) to provide character-level suggestions as users type. While these LLMs offer impressive capabilities, they are computationally intensive, requiring significant processing power and storage. This translates to high operational costs (Cost of Goods Sold - COGS) for every inference. Furthermore, LLMs can be overconfident in their predictions, especially for out-of-distribution (OOD) data, leading to incorrect suggestions that degrade user experience and reduce trust in the application. The core problem is how to leverage the power of LLMs for text prediction while mitigating their high cost and potential for unreliable outputs.
Methods
The authors propose a Model Gatekeeper (GK), a lightweight binary classification model that runs client-side. The GK's job is to predict whether a given input text is likely to result in a useful prediction from the main LLM.
Here's how it works:
- Training Data Generation: The GK is trained using data generated from the large TP model. For each input, the TP model provides a text prediction and a confidence score. A "rendering threshold" is used to label this data: if the TP model's confidence score is below the threshold, the input is labeled as needing a stop (1); otherwise, it's labeled as safe to proceed (0).
- Gatekeeper Model: The GK itself can be either rule-based or model-based.
- Rule-based GK: Employs simple heuristics derived from exploratory data analysis (e.g., input length, last character type, word length).
- Model-based GK: Utilizes a Transformer-based model (specifically, Tiny BERT) for higher accuracy. This approach leverages the fact that the underlying TP model is also Transformer-based.
- Workflow: When a user types a character, the client application first sends the input to the GK. If the GK predicts that the input will lead to an unreliable prediction (outputs 1), the call to the expensive LLM is suppressed. If the GK predicts a reliable prediction (outputs 0), the input is sent to the LLM for suggestion generation.
Datasets
The paper mentions using data from public sources like wiki, books, documents, news, and Technical Support Guides (TSG) for testing and evaluating the TP model. For training the Gatekeeper model, they used input-output pairs generated from the TP model's evaluation on this data. They created training (60%), validation (20%), and test (20%) splits from over 5 million records. The TP model itself was trained using proprietary internal data.
Key Insights and Ideas
- Client-side Filtering is Key: Running a lightweight model on the client side to pre-filter inputs is an effective strategy to avoid unnecessary, expensive server-side LLM calls.
- LLMs are Overconfident: The paper highlights that LLMs can be overly confident in their predictions, especially for out-of-distribution data, leading to false positives that degrade user experience.
- Trade-off Management: The core challenge is balancing COGS reduction with potential losses in character savings. The GK's threshold is crucial for tuning this trade-off.
- Transformer-based GK for Transformer-based LLMs: Using a Transformer architecture (Tiny BERT) for the Gatekeeper model, when the target LLM is also Transformer-based, yields superior performance.
- Rule-based vs. Model-based: While rule-based GKs are simpler, model-based GKs (especially Transformer-based ones) offer significantly better performance in terms of prediction accuracy and efficiency.
Key Results
- COGS Reduction: The proposed Gatekeeper model achieved approximately 46.6% COGS reduction for text prediction.
- Character Savings Loss: This came at the cost of a relatively small 4.5% loss in character savings.
- Suggestion Rate Improvement: The Gatekeeper improved the suggestion rate (percentage of times the server model provided predictions above the confidence threshold) by 73%.
- Model-based GK Performance: A Transformer-based Gatekeeper (Tiny BERT) achieved an AUC of 0.88 on validation and 0.858 on the test set.
- Production Observation: In a production environment, a combined rule+model-based GK achieved approximately 55% COGS saving with less than 1% loss in character saving.
Limitations and Future Work
- Language Specificity: The current work is focused on English text editors and datasets. Future work aims to develop and test COGS-saving Gatekeepers for other languages.
- Model Updates: The current approach requires the GK to be tuned or updated whenever the server-side TP model changes.
- Generality: The authors plan to test and tune Transformer-based Gatekeepers for other large TP models to establish generality.
- Broader Applications: They intend to develop Gatekeepers for other sequence-to-sequence tasks like grammar and sentence correction.
- Continuous Learning: Implementing continuous learning algorithms to update GK models dynamically in response to evolving LLMs is a future direction.
Summary updated