Abstract
Large Language Models (LLMs) like GPT-4 offer state-of-the-art performance but incur substantial operational costs. We present the Online Cost-aware Teacher-Student (OCaTS) framework, a novel approach to reducing LLM API calls by caching responses and training a local student model in real-time. By employing dual reliability criteria—distance-based and entropy-based—OCaTS determines when to trust the student model versus querying the teacher LLM. Our results demonstrate that OCaTS can reduce API calls by up to 34% with a negligible accuracy drop (less than 0.5%), providing a tunable mechanism for cost-efficient AI deployment.
Introduction
For small and medium-sized enterprises (SMEs), the per-token or per-call cost of high-end LLMs creates a significant barrier to scaling customer-facing applications. Observations of production traffic often reveal high redundancy in user queries. While traditional knowledge distillation can create smaller, cheaper models, it typically requires large, pre-labeled datasets that are unavailable at the start of a project.
OCaTS addresses this by operating in an online fashion. It starts with minimal data and learns continuously from teacher responses as real users interact with the system. This framework balances the trade-off between operational expense (OpEx) and model performance through a configurable cost-sensitivity parameter.
The OCaTS Framework
The OCaTS architecture consists of three primary components: a high-performance Teacher (e.g., GPT-4), a lightweight local Student (e.g., k-NN or MLP), and a Cache of previous interactions.
Workflow and Reliability Criteria
When a new query arrives, the system follows a structured decision path:
- Student Prediction: The local student model generates an initial response.
- Reliability Filtering: The system evaluates the prediction using two filters:
- Distance-based: Checks if the input embedding (via MPNet) is sufficiently close to existing examples in the cache.
- Entropy-based: Measures the confidence of the student's probability distribution.
- Routing: If the student is deemed reliable, its response is served immediately. If not, the Teacher is called, the response is returned to the user, and the new data point is added to the Cache to further train the Student.
To quantify the value of this trade-off, we introduce Discounted Accuracy ($\hat{\phi}$), which penalizes the standard accuracy score based on the number of teacher calls and a cost-weighting factor ($\lambda$).
Experimental Evaluation
We evaluated OCaTS on two classification tasks: Banking77 (intent recognition with 77 categories) and Large Movie Review (LMR) (sentiment analysis). We simulated a cold-start environment by providing only 3-10 initial examples per class.
Key Results
OCaTS consistently achieved significant cost savings across different teacher models (GPT-4 and GPT-3.5) and student architectures. On the Banking77 dataset, OCaTS reduced GPT-4 API calls by 34% while maintaining an accuracy of 82.68%, compared to the teacher's baseline of 83.05%.
As shown in Figure 2, as the number of incoming instances increases, the student model matures, leading to a sub-linear growth in teacher calls. By adjusting $\lambda$, developers can tune the system to prioritize either maximum frugality or maximum accuracy.
Conclusion and Future Work
The OCaTS framework provides a practical solution for businesses to leverage LLM capabilities while controlling costs. By treating a local model as a "lazy learner" that matures over time, OCaTS bridges the gap between expensive general-purpose models and specialized local classifiers. Future research will focus on extending this framework to generative tasks such as summarization and developing adaptive policies that automatically adjust reliability thresholds as the student model improves.
Summary updated