Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models

Ilias Stogiannidis, Stavros Vassos, Prodromos Malakasiotis, Ion Androutsopoulos

Abstract

Large Language Models (LLMs) like GPT-4 offer state-of-the-art performance but incur substantial operational costs. We present the Online Cost-aware Teacher-Student (OCaTS) framework, a novel approach to reducing LLM API calls by caching responses and training a local student model in real-time. By employing dual reliability criteria—distance-based and entropy-based—OCaTS determines when to trust the student model versus querying the teacher LLM. Our results demonstrate that OCaTS can reduce API calls by up to 34% with a negligible accuracy drop (less than 0.5%), providing a tunable mechanism for cost-efficient AI deployment.

Introduction

For small and medium-sized enterprises (SMEs), the per-token or per-call cost of high-end LLMs creates a significant barrier to scaling customer-facing applications. Observations of production traffic often reveal high redundancy in user queries. While traditional knowledge distillation can create smaller, cheaper models, it typically requires large, pre-labeled datasets that are unavailable at the start of a project.

OCaTS addresses this by operating in an online fashion. It starts with minimal data and learns continuously from teacher responses as real users interact with the system. This framework balances the trade-off between operational expense (OpEx) and model performance through a configurable cost-sensitivity parameter.

The OCaTS Framework

The OCaTS architecture consists of three primary components: a high-performance Teacher (e.g., GPT-4), a lightweight local Student (e.g., k-NN or MLP), and a Cache of previous interactions.

The OCaTS framework architecture showing the interaction between the Customer, Student model, Cache, and Teacher LLM.

Workflow and Reliability Criteria

When a new query arrives, the system follows a structured decision path:

  1. Student Prediction: The local student model generates an initial response.
  2. Reliability Filtering: The system evaluates the prediction using two filters:
  3. Distance-based: Checks if the input embedding (via MPNet) is sufficiently close to existing examples in the cache.
  4. Entropy-based: Measures the confidence of the student's probability distribution.
  5. Routing: If the student is deemed reliable, its response is served immediately. If not, the Teacher is called, the response is returned to the user, and the new data point is added to the Cache to further train the Student.

To quantify the value of this trade-off, we introduce Discounted Accuracy ($\hat{\phi}$), which penalizes the standard accuracy score based on the number of teacher calls and a cost-weighting factor ($\lambda$).

Experimental Evaluation

We evaluated OCaTS on two classification tasks: Banking77 (intent recognition with 77 categories) and Large Movie Review (LMR) (sentiment analysis). We simulated a cold-start environment by providing only 3-10 initial examples per class.

Key Results

OCaTS consistently achieved significant cost savings across different teacher models (GPT-4 and GPT-3.5) and student architectures. On the Banking77 dataset, OCaTS reduced GPT-4 API calls by 34% while maintaining an accuracy of 82.68%, compared to the teacher's baseline of 83.05%.

Performance comparison of OCaTS vs. GPT-4 across different cost-sensitivity parameters (lambda) showing calls to teacher, accuracy, and discounted accuracy.

As shown in Figure 2, as the number of incoming instances increases, the student model matures, leading to a sub-linear growth in teacher calls. By adjusting $\lambda$, developers can tune the system to prioritize either maximum frugality or maximum accuracy.

Conclusion and Future Work

The OCaTS framework provides a practical solution for businesses to leverage LLM capabilities while controlling costs. By treating a local model as a "lazy learner" that matures over time, OCaTS bridges the gap between expensive general-purpose models and specialized local classifiers. Future research will focus on extending this framework to generative tasks such as summarization and developing adaptive policies that automatically adjust reliability thresholds as the student model improves.

Summary updated

Abstract

Prompting Large Language Models (LLMs) performs impressively in zero- and few-shot settings. Hence, small and medium-sized enterprises (SMEs) that cannot afford the cost of creating large task-specific training datasets, but also the cost of pretraining their own LLMs, are increasingly turning to third-party services that allow them to prompt LLMs. However, such services currently require a payment per call, which becomes a significant operating expense (OpEx). Furthermore, customer inputs are often very similar over time, hence SMEs end-up prompting LLMs with very similar instances. We propose a framework that allows reducing the calls to LLMs by caching previous LLM responses and using them to train a local inexpensive model on the SME side. The framework includes criteria for deciding when to trust the local model or call the LLM, and a methodology to tune the criteria and measure the tradeoff between performance and cost. For experimental purposes, we instantiate our framework with two LLMs, GPT-3.5 or GPT-4, and two inexpensive students, a k-NN classifier or a Multi-Layer Perceptron, using two common business tasks, intent recognition and sentiment analysis. Experimental results indicate that significant OpEx savings can be obtained with only slightly lower performance.
BibTeX
@inproceedings{stogiannidis2023cache,
  author = {Ilias Stogiannidis and Stavros Vassos and Prodromos Malakasiotis and Ion Androutsopoulos},
  title = {{Cache me if you Can: an Online Cost-aware Teacher-Student framework to Reduce the Calls to Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.1000},
}
Figure