LLM Cascade with Multi-Objective Optimal Consideration

Kai Zhang, Liqian Peng, Congchao Wang, Alec Go, Xiaozhong Liu

Abstract

As Large Language Models (LLMs) are increasingly deployed on-device, the trade-off between model size and reasoning capability becomes critical. LLM cascades, which defer complex queries from local models to powerful server-side models, offer a solution but often ignore privacy risks. We present P3Defer, a reinforcement learning-based framework that enables privacy-aware deferral decisions. By integrating Chain-of-Thought (CoT) reasoning and a dedicated private memory module, P3Defer allows models to choose between local execution, server deferral, or masked deferral. Experimental results across multiple benchmarks demonstrate that P3Defer significantly reduces sensitive data leakage while maintaining high task performance and efficiency.

Introduction: The Privacy Blind Spot in LLM Cascades

The proliferation of on-device LLMs, such as Apple Intelligence and Gemini Nano, has popularized the "LLM Cascade" architecture. In this setup, a lightweight local model handles simple queries, while more complex tasks are routed to a massive cloud-based server. Traditionally, the decision to defer is based on the local model's confidence or logit scores.

However, this approach suffers from a significant "privacy blind spot." Standard cascades may inadvertently transmit sensitive personal, medical, or financial information to third-party servers simply because the local model lacks the computational capacity to solve a specific problem. Furthermore, existing privacy-preserving methods that rely on LLM-based query rewriting often compromise the original intent or introduce hallucinations.

Comparison of existing LLM cascades versus the proposed privacy-aware framework.

Methodology: The P3Defer Framework

P3Defer redefines the deferral decision as a multi-objective policy learning problem. The framework consists of three core components:

1. Reinforcement Learning (RL) Agent

Instead of a static threshold, P3Defer employs an agent trained via Proximal Policy Optimization (PPO). The agent evaluates the query and the local model's output to select one of three actions:

  • Accept: Return the local model's answer directly to the user.
  • Defer: Route the raw query to the server model for a higher-quality response.
  • Mask & Defer: Identify sensitive tokens, replace them with masks using a "Private Memory" module, and send the sanitized query to the server.

2. CoT-Enhanced Policy and Private Memory

To improve decision-making, the local model is fine-tuned with Chain-of-Thought (CoT) instructions. This allows the model to "reason" about the query's complexity and privacy implications before the RL agent acts. To ensure robust sanitization, the system utilizes a Private Memory ($\mathcal{M}$) module. Unlike LLM-based rewriting, this module uses Levenshtein distance to identify and mask sensitive tokens, preserving the query's structural integrity while protecting user data.

The detailed architecture of the P3Defer Deferral Module, highlighting the RL agent, reward system, and private memory.

3. Multi-Objective Reward System

The RL agent is optimized using a reward function that balances three competing objectives: quality (maximizing accuracy), cost (minimizing server calls), and privacy (penalizing the leakage of sensitive tokens).

Key Results and Insights

P3Defer demonstrated superior performance over traditional confidence-based baselines across three diverse datasets (GSM8K, MedSum, and EmailSum):

  • Privacy Protection: The "Safe Call Rate" (percentage of privacy-safe server calls) reached 92.6% on GSM8K and 94.6% on EmailSum, a massive improvement over the ~30% achieved by standard methods.
  • Leakage Reduction: The proportion of private tokens sent to the server dropped by 70-75%.
  • Efficiency: The system maintained server-level accuracy while reducing the server call rate to 44.7% on the EmailSum benchmark.

A critical insight from the study is that a model's confidence in its answer does not correlate with the presence of private data. This necessitates the separate, privacy-aware objective introduced by P3Defer.

Conclusion and Future Directions

P3Defer provides a robust middle ground for LLM cascades, moving beyond binary "local vs. server" choices. While the framework introduces higher training costs due to the RL components and currently relies on a pre-processed private memory, it represents a significant step toward truly private on-device AI. Future research will focus on optimizing training efficiency and incorporating latency as an additional objective in the optimization process.

Summary updated

Abstract

Large Language Models (LLMs) have gained significant attention in on-device applications due to their remarkable performance across real-world tasks. However, on-device LLMs often suffer from suboptimal performance due to hardware limitations. A promising solution to this challenge is cascading a weaker local (on-device) LLM with a more powerful server LLM. While existing research on LLM cascade primarily optimizes the performance-cost trade-off, real-world applications impose additional requirements, such as privacy preservation, which remain largely unaddressed. In this work, we move beyond existing confidence- and logit-based LLM cascade methods and propose $\mathbf{P^{3}Defer}$, a novel Chain-of-Thought (CoT)-enhanced \textbf{p}olicy learning framework for \textbf{p}rivacy-\textbf{p}reserved \textbf{defer}ral decision-making. Our approach effectively improves cascade efficiency while mitigating privacy risks. Extensive experiments on three benchmark datasets demonstrate the effectiveness and superiority of $\mathbf{P^{3}Defer}$ over existing methods.
BibTeX
@misc{zhang2024llm,
  author = {Kai Zhang and Liqian Peng and Congchao Wang and Alec Go and Xiaozhong Liu},
  title = {{LLM Cascade with Multi-Objective Optimal Consideration}},
  year = {2024},
  eprint = {2410.08014},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.08014},
}
Figure