Towards Confidential and Efficient LLM Inference with Dual Privacy Protection

Honglan Yu, Yibin Wang, Feifei Dai, Dong Liu, Haihui Fan, Xiaoyan Gu

Abstract

CMIF (Confidential and efficient Model Inference Framework) addresses the critical privacy-efficiency trade-off in cloud-based Large Language Model (LLM) inference. By integrating hardware-based Trusted Execution Environments (TEEs) with a semantically-aware Differential Privacy (DP) mechanism, CMIF protects user data while minimizing the communication overhead typical of secure partitioning. The framework deploys the sensitive embedding layer within a client-side TEE and utilizes a Report-Noisy-Max (RNM) sanitization process to replace sensitive tokens with semantically similar alternatives. Evaluations across benchmarks such as SST-2, QNLI, and IFEval demonstrate that CMIF maintains high utility and instruction-following capabilities with significantly lower latency than traditional TEE-partitioning methods.

Introduction

Deploying Large Language Models (LLMs) on cloud servers raises significant privacy concerns, as user queries often contain sensitive personal information. Current solutions generally fall into two categories, both with major drawbacks. TEE-based partitioning attempts to run sensitive model components in secure CPU enclaves (e.g., Intel SGX) while offloading linear operations to GPUs. However, because nonlinear layers like Softmax must remain inside the TEE for security, constant data transfers between the CPU and GPU lead to massive latency. Alternatively, standard Differential Privacy (DP) methods add random noise to data, which often destroys the semantic meaning of text, resulting in incoherent model outputs. CMIF seeks a middle ground that provides hardware-level security and data privacy without sacrificing inference speed or model utility.

Methodology

CMIF introduces a dual-protection strategy designed to balance security and performance.

TEE-Shielded Deployment

Instead of partitioning every layer, CMIF deploys only the embedding layer (which maps tokens to vectors) inside a client-side TEE (Intel SGX2). The remainder of the model, including decoder and output layers, resides on a high-performance GPU server. This architecture minimizes the frequency of context switching and data transfers between the TEE and the GPU, addressing the primary bottleneck of previous partitioning methods.

CMIF Architecture Overview: The client-side TEE handles sanitization and embedding, while the server-side GPU processes the bulk of the model.

Report-Noisy-Max (RNM) Sanitization

To protect data sent to the server, CMIF employs an optimized RNM mechanism for semantic sanitization:

  1. Sensitivity Identification: Low-frequency words are identified as "sensitive."
  2. Distance Function: For each sensitive word, the system identifies the top-$k$ most semantically similar words in the vocabulary using cosine similarity.
  3. Mapping Function: The original word is replaced with a candidate by adding Laplace noise to similarity scores and selecting the one with the highest noisy score. This ensures replacements are semantically related while providing mathematical $\epsilon$-Differential Privacy guarantees.
  4. Probabilistic Replacement: Non-sensitive words are subject to a replacement probability $p$, further obfuscating the input sequence.

Evaluation and Results

The framework was evaluated using BERT, Llama2-7B, and Llama3-8B across several benchmarks, including SST-2 and QNLI (classification), DialogSUM (summarization), and IFEval (instruction following).

Superior Utility

On the SST-2 and QNLI datasets, the RNM mechanism consistently outperformed existing DP methods like SANTEXT+ and CUSTEXT+. For instance, at a privacy level of $\epsilon=1$, CMIF achieved approximately 82.6% accuracy on SST-2, compared to roughly 75% for competing methods.

Comparison of RNM with existing DP methods (CUSTEXT, SANTEXT+) across different privacy budgets.

Instruction Following and Efficiency

On the IFEval benchmark, Llama3-8B's performance dropped by a negligible 0.15% when using RNM, proving the method preserves complex reasoning capabilities. In terms of efficiency, while CMIF adds an overhead of 28-38% compared to a non-private baseline (primarily due to SGX2 encryption), the sanitization algorithm itself accounts for less than 1% of total inference time. This represents a significant improvement over traditional TEE-partitioning methods that suffer from frequent TEE-GPU data transfers.

Discussion and Insights

A key finding is that replacing sensitive words with semantically similar ones, rather than adding Gaussian noise to vectors, allows the model to maintain the "gist" of a query. Furthermore, researchers observed that Llama3-8B is more robust to DP perturbations than Llama2-7B, likely due to its 4x larger vocabulary and more extensive pre-training. By moving the privacy boundary to the embedding layer, the communication overhead shifts from being a function of the number of layers to a fixed cost associated with the initial TEE encryption/decryption.

Limitations

The majority of CMIF's latency still stems from the inherent overhead of Intel SGX2. Additionally, the current evaluation assumes a "semi-honest" threat model and does not account for potential side-channel attacks on TEEs. Future work will explore hardware-accelerated TEEs and further hyperparameter tuning for specific application domains.

Summary updated

BibTeX
@inproceedings{yu2025towards,
  author = {Honglan Yu and Yibin Wang and Feifei Dai and Dong Liu and Haihui Fan and Xiaoyan Gu},
  title = {{Towards Confidential and Efficient LLM Inference with Dual Privacy Protection}},
  booktitle = {DASFAA},
  year = {2025},
  doi = {10.1007/978-981-95-4155-3\_42},
}
Figure