Abstract
Large Language Models (LLMs) are increasingly used for simple, repetitive tasks due to their zero-shot capabilities, but they incur significant operational costs and latency. We present Poodle, a framework implementing Just-in-Time Model Replacement (JITR). Poodle automatically identifies recurring tasks, generates labeled datasets via LLM knowledge distillation, searches for optimal small-scale base models, and replaces the expensive LLM with a specialized surrogate. Our evaluation shows that Poodle can reduce inference costs by over 80x and increase throughput by 20x while maintaining LLM-level accuracy, paying for its own development costs in as few as 10,000 requests.
Introduction
Organizations often default to massive LLMs for tasks like sentiment analysis or data extraction to avoid the engineering overhead of data collection and model training. While this offers "zero development cost," it imposes a high "LLM Tax" in the form of massive operational costs, high latency, and significant energy consumption. Poodle addresses this by bridging the gap between the ease of LLM deployment and the efficiency of specialized models through an automated transition process.
The JITR Framework
Poodle is inspired by just-in-time compilation in programming languages. The workflow consists of four automated stages:
- Task Identification & Dataset Generation: Poodle monitors incoming requests using a "wrapper prompt." This instruction tells the LLM to perform the user's task while simultaneously categorizing the task type. This allows the system to build a labeled dataset of request-response pairs automatically, solving the "cold start" problem of metadata.
- Base Model Search: Instead of training from scratch, Poodle searches model stores (e.g., HuggingFace) for a promising "base model" that fits the task and hardware constraints.
- Surrogate Model Development: The selected base model is fine-tuned using the labels generated by the LLM, a process known as knowledge distillation.
- Deployment & Monitoring: Once the surrogate model reaches sufficient accuracy, Poodle offers the user a switch. It then continuously monitors the surrogate's performance against the LLM to detect potential data drift.
Experimental Evaluation
We evaluated Poodle using the IMDB Movie Reviews dataset for sentiment classification and the HuggingFace Model Hub for model search.
Key Results
- Massive Cost Reduction: For 1 million requests, Poodle was 60x cheaper than using GPT-4.1 and 82x cheaper than Llama 405B.
- Throughput Gains: In self-hosted scenarios, a fine-tuned BERT surrogate processed items 19.6x faster than a Llama-2-7B model.
- Competitive Accuracy: Small surrogate models (like BERT) matched or slightly exceeded the accuracy of flagship LLMs (~0.90 accuracy) on classification tasks.
- Fast Break-Even: The initial overhead of wrapper prompts and fine-tuning is quickly amortized. For large models like Llama 405B, Poodle pays for its own development costs in fewer than 10,000 requests.
Discussion and Future Work
While Poodle demonstrates significant gains for classification tasks, several challenges remain. The current prototype focuses on classification; extending JITR to complex generative tasks or multi-step reasoning is a key area for future research. Additionally, the "wrapper prompt" strategy increases initial token counts, and scaling model search to the millions of models available online requires more advanced indexing and similarity metrics. Finally, robust distribution shift detection is essential to ensure surrogate models remain accurate as user data evolves.
Summary updated