TL;DR
CENTAUR is a novel privacy-preserving framework for Transformer inference that breaks the "impossible trinity" of balancing privacy, efficiency, and performance. By strategically combining random permutations with Secure Multi-Party Computation (SMPC), it achieves plaintext-level accuracy and a 5x to 30x inference speedup over traditional cryptographic methods while robustly protecting both model parameters and user data.
What Problem Does This Paper Address?
Deploying large Transformer models (like BERT or GPT) on cloud platforms introduces severe privacy risks: model developers risk losing proprietary weights, and users risk exposing sensitive prompts. Existing Privacy-Preserving Transformer Inference (PPTI) frameworks force a compromise across three dimensions—the "impossible trinity":
- Privacy: Secure Multi-Party Computation (SMPC) provides strong mathematical guarantees but suffers from massive communication overhead.
- Efficiency: Permutation-based methods bypass heavy cryptography by shuffling data, but they leak intermediate states.
- Performance: To make SMPC practical, some frameworks replace complex non-linear functions (like Softmax or GeLU) with approximations, which degrades model accuracy.
As shown in Figure 1, previous methods could only achieve two out of three. Furthermore, the authors demonstrate that pure permutation-based methods are highly vulnerable to modern Data Reconstruction Attacks (DRAs), which can reverse-engineer original text prompts from intermediate model layers.
Methods
CENTAUR introduces a hybrid architecture that splits the privacy workload between random permutations and 2-out-of-2 secret sharing (SMPC) across a three-party semi-honest model consisting of a Model Developer ($P_0$), a Cloud Platform ($P_1$), and a Client ($P_2$).
- Linear Layers: Instead of encrypting everything, the model weights are kept in a plaintext, randomly permuted state ($\Theta'$), while the user's inference data is split into cryptographic secret shares ($[X]_0, [X]_1$). This allows matrix multiplications to be computed using a communication-free protocol ($\Pi_{ScalMul}$), completely eliminating the network bottleneck typical of SMPC linear layers.
- Non-Linear Layers: For complex operations like Softmax, GeLU, and LayerNorm, CENTAUR temporarily transitions the secret-shared data into a randomly permuted plaintext state on the cloud platform. Because the data is randomly shuffled across high feature dimensions, the cloud provider can safely perform exact, plaintext element-wise computations without learning the underlying information.
Datasets
The authors evaluated CENTAUR across several standard benchmarks:
- GLUE Benchmark: Five natural language understanding (NLU) tasks (RTE, CoLA, STS-B, MRPC, QNLI) evaluated on BERT-Base and BERT-Large.
- Wikitext Collection: Wikitext-2 and Wikitext-103 datasets for natural language generation (NLG) tasks evaluated on GPT-2-Base and GPT-2-Large.
- CNN-DailyMail: Used as an auxiliary dataset to train the inversion models for simulating realistic data reconstruction attacks.
- LLaMA-7B Evaluation: Used to demonstrate generalizability to modern large language models.
Key Insights and Ideas
- The Power of Hybridization: Pure cryptography is too slow, and pure obfuscation is insecure. By applying permutations to the model structure and secret sharing to the user data, CENTAUR cancels out the weaknesses of both.
- Breaking Inversion Attacks via Feature Shuffling: Data reconstruction attacks rely on mapping intermediate representations back to a coherent vocabulary space. Randomly shuffling features across large hidden dimensions (e.g., 768 or 1280) completely disrupts this inverse mapping function.
- Zero-Overhead Accuracy: Because non-linear functions are computed exactly on permuted plaintexts rather than approximated via polynomials, the framework requires zero retraining or knowledge distillation to maintain accuracy.
Key Results
Security Against Reconstruction Attacks
When subjected to state-of-the-art reconstruction attacks (SIP, EIA, BRE), unprotected permutation methods allowed attackers to recover text with high fidelity. Under CENTAUR, the attacker's ROUGE-L recovery scores plummeted to baseline random noise levels (frequently under 5%).
Efficiency and Speedups
CENTAUR dramatically outperformed existing SMPC baselines (PUMA, MPCFormer, SecFormer) in both Local Area Network (LAN) and Wide Area Network (WAN) settings:
- BERT-Large: Achieved a 5.1x to 24.2x speedup in LAN and a 6.3x to 30.4x speedup in WAN.
- GPT-2-Large: Achieved a 5.0x to 26.9x speedup in LAN and a 5.8x to 28.4x speedup in WAN.
This efficiency gain is directly tied to the massive reduction in communication volume achieved by bypassing cryptographic matrix multiplications.
Performance Retention
Unlike competing frameworks that suffer from accuracy drops, CENTAUR achieved 100% identical performance to standard plaintext inference across all GLUE and Wikitext tasks.
Limitations and Future Work
While highly practical, CENTAUR relies on empirical security guarantees for its permutation mechanisms rather than strict, provable theoretical security.
Additionally, the current implementation does not incorporate other common LLM optimizations such as quantization or KV-caching. Because KV-caching involves operations like top-k selection and token aggregation that are inherently unfriendly to secret sharing, integrating these optimizations securely remains an open challenge for future work.
Summary updated