Abstract
The deployment of large-scale foundation models on edge devices is hindered by computational inefficiency and privacy concerns regarding sensitive user data and model intellectual property. This paper presents PrivTuner, a novel framework that establishes a "Privacy-Preserving Parameter-Efficient Fine-Tuning" (P3EFT) paradigm. By combining Low-Rank Adaptation (LoRA) with Fully Homomorphic Encryption (FHE) and a joint resource allocation algorithm, PrivTuner allows mobile devices to fine-tune models using encrypted data. Our evaluation on the GLUE benchmark shows that PrivTuner achieves competitive accuracy with minimal performance degradation while optimizing energy consumption in wireless networks.
1. Introduction
Foundation models such as GPT and BERT have revolutionized natural language processing but face significant barriers to edge deployment. Specifically, the Efficiency Gap makes full-parameter fine-tuning prohibitive for mobile hardware, while the Privacy Gap prevents the sharing of raw data or proprietary model weights. While techniques like LoRA address efficiency, they typically require plaintext access. Conversely, privacy-preserving methods often incur unsustainable computational overhead.
PrivTuner bridges these gaps by defining the P3EFT space—the intersection of Parameter-Efficient Fine-Tuning (PEFT) and Privacy-Preserving Fine-Tuning (PPFT), as illustrated in the conceptual diagrams above.
2. Methodology: The PrivTuner Architecture
PrivTuner integrates four core components to achieve secure and efficient fine-tuning:
- Low-Rank Adaptation (LoRA): To minimize trainable parameters, PrivTuner updates only two small matrices ($A$ and $B$) rather than the entire weight matrix. This reduces both communication and computation costs.
- Fully Homomorphic Encryption (FHE): Utilizing the CKKS scheme, PrivTuner enables the server to perform mathematical operations on encrypted data. This ensures that user data remains confidential throughout the fine-tuning process.
- HE-Friendly BERT-Tiny: Standard Transformer architectures rely on non-linear functions (e.g., Softmax, GeLU) incompatible with FHE. PrivTuner approximates these using Maclaurin series and Chebyshev polynomials, creating a "crypto-friendly" model variant.
- Joint Resource Allocation (RA): A specialized algorithm optimizes bandwidth, transmission power, CPU frequency, and the FHE polynomial degree ($\lambda$). This ensures the system meets privacy and latency constraints while minimizing total energy consumption.
3. Experimental Evaluation
The framework was evaluated using the GLUE (General Language Understanding Evaluation) benchmark, focusing on tasks such as sentiment analysis (SST-2), paraphrase detection (MRPC), and linguistic acceptability (CoLA).
3.1 Accuracy and Performance
Experimental results indicate that the polynomial approximations in the FHE-enabled BERT-Tiny result in a negligible accuracy drop. For instance, on the SST-2 task, the FHE version achieved 79.0% accuracy, compared to 82.3% for the plaintext baseline. On average, the accuracy degradation across tasks was approximately 3%.
3.2 Efficiency and Latency
While client-side encryption and decryption are efficient (taking less than one second), server-side computation on ciphertexts remains the primary bottleneck. A single encrypted inference pass takes approximately 163 seconds. However, the joint RA algorithm significantly reduces energy consumption compared to standard allocation benchmarks by dynamically adjusting parameters based on channel conditions.
4. Discussion and Key Insights
- Privacy-Energy Trade-off: The security level in FHE is directly tied to the polynomial degree. PrivTuner treats the security level as a tunable variable, allowing for energy savings when maximum privacy is not required.
- Viability of FHE-based Deep Learning: The small accuracy drop from polynomial approximations proves that FHE is a viable path for secure deep learning tasks, provided the model architecture is appropriately adapted.
- Security Guarantees: PrivTuner provides measurable security (up to 277 bits) against standard cryptographic attacks (uSVP, BDD), offering formal protection for user data.
5. Limitations and Future Directions
Despite its promise, PrivTuner faces challenges in latency. Future work will explore hardware acceleration via GPUs and FPGAs to reduce encrypted inference times to sub-second levels. Additionally, scaling P3EFT to larger models and addressing malicious (non-honest-but-curious) threat models remain critical areas for research. Finally, moving label-related loss calculations to the server while maintaining privacy could further streamline the fine-tuning pipeline.