CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference

Jinglong Luo, Guanzhong Chen, Yehong Zhang, Shiyu Liu, Hui Wang, Yue Yu, Xun Zhou, Yuan Qi, Zenglin Xu

TL;DR

CENTAUR is a novel privacy-preserving framework for Transformer inference that breaks the "impossible trinity" of balancing privacy, efficiency, and performance. By strategically combining random permutations with Secure Multi-Party Computation (SMPC), it achieves plaintext-level accuracy and a 5x to 30x inference speedup over traditional cryptographic methods while robustly protecting both model parameters and user data.

What Problem Does This Paper Address?

Deploying large Transformer models (like BERT or GPT) on cloud platforms introduces severe privacy risks: model developers risk losing proprietary weights, and users risk exposing sensitive prompts. Existing Privacy-Preserving Transformer Inference (PPTI) frameworks force a compromise across three dimensions—the "impossible trinity":

  1. Privacy: Secure Multi-Party Computation (SMPC) provides strong mathematical guarantees but suffers from massive communication overhead.
  2. Efficiency: Permutation-based methods bypass heavy cryptography by shuffling data, but they leak intermediate states.
  3. Performance: To make SMPC practical, some frameworks replace complex non-linear functions (like Softmax or GeLU) with approximations, which degrades model accuracy.

As shown in Figure 1, previous methods could only achieve two out of three. Furthermore, the authors demonstrate that pure permutation-based methods are highly vulnerable to modern Data Reconstruction Attacks (DRAs), which can reverse-engineer original text prompts from intermediate model layers.

Figure 1: Overview of CENTAUR and Other PPTI Frameworks.

Methods

CENTAUR introduces a hybrid architecture that splits the privacy workload between random permutations and 2-out-of-2 secret sharing (SMPC) across a three-party semi-honest model consisting of a Model Developer ($P_0$), a Cloud Platform ($P_1$), and a Client ($P_2$).

Figure 3: High-level Workflow of CENTAUR.
  • Linear Layers: Instead of encrypting everything, the model weights are kept in a plaintext, randomly permuted state ($\Theta'$), while the user's inference data is split into cryptographic secret shares ($[X]_0, [X]_1$). This allows matrix multiplications to be computed using a communication-free protocol ($\Pi_{ScalMul}$), completely eliminating the network bottleneck typical of SMPC linear layers.
  • Non-Linear Layers: For complex operations like Softmax, GeLU, and LayerNorm, CENTAUR temporarily transitions the secret-shared data into a randomly permuted plaintext state on the cloud platform. Because the data is randomly shuffled across high feature dimensions, the cloud provider can safely perform exact, plaintext element-wise computations without learning the underlying information.

Datasets

The authors evaluated CENTAUR across several standard benchmarks:

  • GLUE Benchmark: Five natural language understanding (NLU) tasks (RTE, CoLA, STS-B, MRPC, QNLI) evaluated on BERT-Base and BERT-Large.
  • Wikitext Collection: Wikitext-2 and Wikitext-103 datasets for natural language generation (NLG) tasks evaluated on GPT-2-Base and GPT-2-Large.
  • CNN-DailyMail: Used as an auxiliary dataset to train the inversion models for simulating realistic data reconstruction attacks.
  • LLaMA-7B Evaluation: Used to demonstrate generalizability to modern large language models.

Key Insights and Ideas

  • The Power of Hybridization: Pure cryptography is too slow, and pure obfuscation is insecure. By applying permutations to the model structure and secret sharing to the user data, CENTAUR cancels out the weaknesses of both.
  • Breaking Inversion Attacks via Feature Shuffling: Data reconstruction attacks rely on mapping intermediate representations back to a coherent vocabulary space. Randomly shuffling features across large hidden dimensions (e.g., 768 or 1280) completely disrupts this inverse mapping function.
  • Zero-Overhead Accuracy: Because non-linear functions are computed exactly on permuted plaintexts rather than approximated via polynomials, the framework requires zero retraining or knowledge distillation to maintain accuracy.

Key Results

Security Against Reconstruction Attacks

When subjected to state-of-the-art reconstruction attacks (SIP, EIA, BRE), unprotected permutation methods allowed attackers to recover text with high fidelity. Under CENTAUR, the attacker's ROUGE-L recovery scores plummeted to baseline random noise levels (frequently under 5%).

Figure 2: Two examples of recovering private inference input data through attacks on intermediate results. On the left are the real data, and on the right are the data reconstructed using data reconstruction attacks. Green indicates complete recovery, while orange signifies ap- proximate recovery.

Efficiency and Speedups

CENTAUR dramatically outperformed existing SMPC baselines (PUMA, MPCFormer, SecFormer) in both Local Area Network (LAN) and Wide Area Network (WAN) settings:

  • BERT-Large: Achieved a 5.1x to 24.2x speedup in LAN and a 6.3x to 30.4x speedup in WAN.
  • GPT-2-Large: Achieved a 5.0x to 26.9x speedup in LAN and a 5.8x to 28.4x speedup in WAN.

This efficiency gain is directly tied to the massive reduction in communication volume achieved by bypassing cryptographic matrix multiplications.

Performance Retention

Unlike competing frameworks that suffer from accuracy drops, CENTAUR achieved 100% identical performance to standard plaintext inference across all GLUE and Wikitext tasks.

Limitations and Future Work

While highly practical, CENTAUR relies on empirical security guarantees for its permutation mechanisms rather than strict, provable theoretical security.

Additionally, the current implementation does not incorporate other common LLM optimizations such as quantization or KV-caching. Because KV-caching involves operations like top-k selection and token aggregation that are inherently unfriendly to secret sharing, integrating these optimizations securely remains an open challenge for future work.

Summary updated

Abstract

With the growing deployment of pre-trained models like Transformers on cloud platforms, privacy concerns about model parameters and inference data are intensifying. Existing Privacy-Preserving Transformer Inference (PPTI) frameworks face the "impossible trinity" of balancing privacy, efficiency, and performance: Secure Multi-Party Computation (SMPC)-based approaches ensure strong privacy but suffer from high computational overhead and performance losses; Conversely, permutation-based methods achieve near-plaintext efficiency and accuracy but compromise privacy by exposing sensitive model parameters and intermediate results. Bridging this gap with a single approach presents substantial challenges, motivating the introduction of CENTAUR, a groundbreaking PPTI framework that seamlessly integrates random permutations and SMPC to address the "impossible trinity". By designing efficient PPTI algorithms tailored to the structural properties of Transformer models, CENTAUR achieves an unprecedented balance among privacy, efficiency, and performance. Our experiments demonstrate CENTAUR's ability to resist diverse data reconstruction attacks, achieve plaintext-level inference accuracy, and boost inference speed by 5.0-30.4 times, unlocking new possibilities for secure and efficient AI deployment.
BibTeX
@inproceedings{luo2025centaur,
  author = {Jinglong Luo and Guanzhong Chen and Yehong Zhang and Shiyu Liu and Hui Wang and Yue Yu and Xun Zhou and Yuan Qi and Zenglin Xu},
  title = {{CENTAUR: Bridging the Impossible Trinity of Privacy, Efficiency, and Performance in Privacy-Preserving Transformer Inference}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure