TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses

Muhammad Taha Cheema, Abeer Aamir, Khawaja Gul Muhammad, Naveed Anwar Bhatti, Ihsan Ayyub Qazi, Zafar Ayyub QaziarXiv 2025 · DOI · arXiv

TL;DR

Caching responses from Large Language Models (LLMs) is crucial for reducing costs and latency, but traditional methods struggle with the nuanced, personalized nature of chatbot interactions. TweakLLM introduces a novel two-tier architecture that combines semantic similarity search with a lightweight LLM to dynamically adapt cached responses, significantly improving cache effectiveness without sacrificing response quality.

What Problem Does This Paper Address?

Modern LLMs are incredibly powerful but also computationally expensive and slow to run. To mitigate this, researchers are exploring caching strategies, where previously generated responses to similar queries are reused. However, this approach faces a significant challenge: user queries in conversational AI are highly personalized and context-dependent. Even minor wording changes can alter user intent, making it risky to simply return a cached response based on semantic similarity alone. Traditional semantic caching methods often fall into a precision-recall trap: either they accept too many inaccurate "false positive" cache hits at low similarity thresholds, or they achieve high precision only by setting very high thresholds, leading to negligible cache utilization. This paper tackles the problem of making LLM caching effective and reliable for dynamic, personalized conversations.

Methods

TweakLLM proposes a two-tier architecture:

  1. Semantic Cache Lookup: When a user query arrives, it's first embedded and compared against a vector database of cached queries using cosine similarity. If the similarity exceeds a predefined threshold, a candidate cached response is retrieved.
  2. Dynamic Adjustment (Lightweight LLM Refinement): Instead of returning the cached response verbatim, a smaller, cost-effective LLM (referred to as "Small LLM") is employed. This Small LLM is prompted to tailor the cached response to the nuances of the incoming prompt, ensuring relevance, accuracy, and clarity. This process aims to adapt the high-quality cached response to the specific user's intent.

If the initial semantic similarity is below the threshold, the query is routed directly to a powerful "Big LLM" for a fresh response, which is then added to the cache.

Datasets

The authors evaluated TweakLLM on several real-world datasets:

  • Question Pairs Dataset (Quora): A dataset of over 400,000 question pairs, useful for identifying similar questions.
  • allenai/WildChat-1M: A dataset of 650K conversations with OpenAI's GPT-3.5 and GPT-4.
  • Imsys/lmsys-chat-1m: A dataset of 1 million user-chatbot conversations from various LLMs.

For the LMSYS dataset, they filtered for English conversations and used only the first user query from each conversation.

Key Insights and Ideas

  • Semantic Similarity Alone is Insufficient: The paper strongly demonstrates that relying solely on semantic similarity for LLM caching leads to an unacceptable trade-off between precision and recall in real-world conversational scenarios.
  • The Power of Dynamic Tailoring: By using a lightweight LLM to adapt cached responses, TweakLLM can significantly expand the utility of the cache. This refinement step allows for the reuse of high-quality responses while ensuring they are precisely tailored to the user's current query.
  • Bridging the Gap with Small LLMs: TweakLLM effectively leverages the strengths of smaller LLMs for response refinement, offering a cost-effective way to improve cache efficiency without compromising the quality typically associated with larger, more expensive models.
  • Handling Nuance and Ambiguity: TweakLLM is particularly effective in scenarios where traditional caching fails, such as queries with similar phrasing but opposite intents (e.g., "Why is X good?" vs. "Why is X bad?").

Key Results

  • User Study: Participants rated TweakLLM's refined responses as equivalently satisfactory to direct Big LLM responses. In side-by-side comparisons, users voted for "Draw" or "Small LLM" more often than "Big LLM" exclusively.
  • LLM-as-Evaluator: Automated evaluations showed that as the similarity threshold increased, Small LLM-tweaked responses were increasingly judged as on par or better than direct Big LLM generations. This trend held across both curated (Quora) and more free-form (LMSYS) datasets.
  • Cost Savings: By leveraging TweakLLM, inference costs for WildChat and LMSYS datasets could be reduced to 61% and 35% of the original cost, respectively, due to increased cache hit rates.
  • Precision-Recall Trade-off: Traditional caching (GPTCache) showed that achieving 97% precision required a recall of only 0.2, highlighting its limitations. TweakLLM aims to improve recall without sacrificing precision.

Limitations and Future Work

  • Multi-turn Conversations: The current evaluation focused on single-turn queries. Future work should assess TweakLLM's performance in multi-turn conversations, which is crucial for production chatbots.
  • Query Type Analysis: The paper did not analyze performance variations across different query types (e.g., factual, advice-based).
  • Productionization Aspects: Content moderation, temporal filtering, and cache eviction policies were not explored but are important for production deployment.
  • Automated Evaluation Diversity: Current automated evaluations rely on GPT-40 referees; diversifying this with other models is suggested.
  • Latency Gains and Cost Improvements: Quantifying expected latency gains and precise cost improvements beyond API costs is a future direction.
  • Component Swapping: The modular architecture allows for evaluating different components, such as swapping embedding models.

Summary updated

BibTeX
@misc{cheema2025tweakllm,
  author = {Muhammad Taha Cheema and Abeer Aamir and Khawaja Gul Muhammad and Naveed Anwar Bhatti and Ihsan Ayyub Qazi and Zafar Ayyub Qazi},
  title = {{TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses}},
  year = {2025},
  eprint = {2507.23674},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.23674},
}
Figure