Abstract
As Large Language Models (LLMs) are increasingly deployed on-device, the trade-off between model size and reasoning capability becomes critical. LLM cascades, which defer complex queries from local models to powerful server-side models, offer a solution but often ignore privacy risks. We present P3Defer, a reinforcement learning-based framework that enables privacy-aware deferral decisions. By integrating Chain-of-Thought (CoT) reasoning and a dedicated private memory module, P3Defer allows models to choose between local execution, server deferral, or masked deferral. Experimental results across multiple benchmarks demonstrate that P3Defer significantly reduces sensitive data leakage while maintaining high task performance and efficiency.
Introduction: The Privacy Blind Spot in LLM Cascades
The proliferation of on-device LLMs, such as Apple Intelligence and Gemini Nano, has popularized the "LLM Cascade" architecture. In this setup, a lightweight local model handles simple queries, while more complex tasks are routed to a massive cloud-based server. Traditionally, the decision to defer is based on the local model's confidence or logit scores.
However, this approach suffers from a significant "privacy blind spot." Standard cascades may inadvertently transmit sensitive personal, medical, or financial information to third-party servers simply because the local model lacks the computational capacity to solve a specific problem. Furthermore, existing privacy-preserving methods that rely on LLM-based query rewriting often compromise the original intent or introduce hallucinations.
Methodology: The P3Defer Framework
P3Defer redefines the deferral decision as a multi-objective policy learning problem. The framework consists of three core components:
1. Reinforcement Learning (RL) Agent
Instead of a static threshold, P3Defer employs an agent trained via Proximal Policy Optimization (PPO). The agent evaluates the query and the local model's output to select one of three actions:
- Accept: Return the local model's answer directly to the user.
- Defer: Route the raw query to the server model for a higher-quality response.
- Mask & Defer: Identify sensitive tokens, replace them with masks using a "Private Memory" module, and send the sanitized query to the server.
2. CoT-Enhanced Policy and Private Memory
To improve decision-making, the local model is fine-tuned with Chain-of-Thought (CoT) instructions. This allows the model to "reason" about the query's complexity and privacy implications before the RL agent acts. To ensure robust sanitization, the system utilizes a Private Memory ($\mathcal{M}$) module. Unlike LLM-based rewriting, this module uses Levenshtein distance to identify and mask sensitive tokens, preserving the query's structural integrity while protecting user data.
3. Multi-Objective Reward System
The RL agent is optimized using a reward function that balances three competing objectives: quality (maximizing accuracy), cost (minimizing server calls), and privacy (penalizing the leakage of sensitive tokens).
Key Results and Insights
P3Defer demonstrated superior performance over traditional confidence-based baselines across three diverse datasets (GSM8K, MedSum, and EmailSum):
- Privacy Protection: The "Safe Call Rate" (percentage of privacy-safe server calls) reached 92.6% on GSM8K and 94.6% on EmailSum, a massive improvement over the ~30% achieved by standard methods.
- Leakage Reduction: The proportion of private tokens sent to the server dropped by 70-75%.
- Efficiency: The system maintained server-level accuracy while reducing the server call rate to 44.7% on the EmailSum benchmark.
A critical insight from the study is that a model's confidence in its answer does not correlate with the presence of private data. This necessitates the separate, privacy-aware objective introduced by P3Defer.
Conclusion and Future Directions
P3Defer provides a robust middle ground for LLM cascades, moving beyond binary "local vs. server" choices. While the framework introduces higher training costs due to the RL components and currently relies on a pre-processed private memory, it represents a significant step toward truly private on-device AI. Future research will focus on optimizing training efficiency and incorporating latency as an additional objective in the optimization process.
Summary updated