Tempo: Confidentiality Preservation in Cloud-Based Neural Network Training

Rongwu Xu, Zhixuan Fang

TL;DR

Tempo is a novel cloud-based deep learning system that uniquely combines Trusted Execution Environments (TEEs) with distributed GPUs to enable private and efficient neural network training. It addresses the critical challenge of protecting both input data and trained models from untrusted cloud providers, achieving significant speedups over existing secure methods.

What Problem Does This Paper Address?

The increasing reliance on cloud platforms for deep learning (DL) training offers cost-effectiveness and computational power, but it introduces significant privacy risks. Cloud Service Providers (CSPs) can be untrustworthy, potentially exposing sensitive training data and the intellectual property embodied in trained models. While Trusted Execution Environments (TEEs) offer a way to secure computations, they are typically too slow for practical DL training without hardware acceleration. Existing solutions that offload computations to GPUs often fail to simultaneously protect both input data and the model itself, or they are limited to inference tasks. This paper tackles the gap of providing a comprehensive solution for both input and model privacy during training in a cloud environment, leveraging the strengths of both TEEs and GPUs.

Methods

Tempo's core innovation lies in its hybrid approach:

  • TEE as a Trusted Master: A TEE (specifically Intel SGX) acts as a secure "master" node. It handles sensitive operations like key generation, encryption, and decryption, ensuring that the model parameters and input data remain confidential within its protected enclave.
  • Distributed GPUs for Computation: Untrusted worker nodes equipped with GPUs perform the computationally intensive matrix multiplications required for DL training.
  • MM-Obfuscation Algorithm: To protect data during offloading, Tempo introduces a novel "permutation-based secure matrix multiplication" algorithm. This algorithm blinds both the model weights (W) and input features (X) before they are sent to the GPUs. This obfuscation is designed to be reversible within the TEE and to maintain the correctness of the linear operations.
  • Optimized Backpropagation: Tempo employs a key-shift mechanism and reuses encrypted results from the forward pass to reduce the number of encryption/decryption operations during backpropagation by 50%, significantly speeding up the training process.
  • Hybrid Parallelism: For distributed training across multiple workers, Tempo utilizes a combination of Data Parallelism (DP), Tensor Model Parallelism (TP), and Pipeline Parallelism (PP), dynamically choosing the best strategy based on the layer type (e.g., TP for parameter-heavy FC layers, DP for data-heavy convolutional layers).

Datasets

The paper evaluates Tempo using the CIFAR-10 dataset for image classification tasks.

Key Insights and Ideas

  • TEE + GPU Synergy: The paper demonstrates that combining TEEs for security and GPUs for performance is not only feasible but also highly effective for private DL training, overcoming the limitations of each technology in isolation.
  • Obfuscation for Linear Operations: The MM-obfuscation algorithm is crucial. It cleverly blinds matrices in a way that allows for secure offloading of linear algebra, which forms the backbone of neural networks, without compromising privacy.
  • Reduced Encryption Overhead: The optimization for backpropagation is a key practical insight. By reusing computations and strategically shifting keys, Tempo significantly cuts down on the expensive encryption/decryption steps, which are often bottlenecks in secure DL systems.
  • Model Privacy is Achievable: Unlike many prior works focusing solely on input privacy, Tempo explicitly tackles model privacy, recognizing the significant IP value of trained models.

Key Results

  • Training Speedup: Tempo achieves a significant training speed-up of 4.37x (local setup) and 3.95x (network setup) compared to pure TEE baselines, without compromising model accuracy.
  • Accuracy Preservation: Tempo maintains comparable accuracy to baseline methods on both ResNet50 and Vision Transformer (ViT) models trained on CIFAR-10.
  • Efficiency: For ResNet50, Tempo-l (1 GPU) achieved a maximal training speed-up of 4.92x over the secure baseline, and for ViT, it was 3.83x.
  • Privacy Guarantees: Theoretical analysis and empirical evaluations (using Mutual Information) suggest that Tempo provides strong privacy against model theft and input leakage attacks.

Limitations and Future Work

  • Network Bandwidth Dependency: Tempo-n (networked prototype) performance degrades significantly with lower network bandwidths, becoming impractical below approximately 129.82 MBps.
  • TEE Limitations: The paper acknowledges that SGX side-channel attacks are beyond the scope of this work.
  • Future Work: The authors suggest exploring how Tempo can collaborate with other distributed training algorithms to further boost performance, indicating potential for more advanced hybrid parallelism strategies. They also highlight the need for further research into mitigating black-box attacks on inference APIs, even though Tempo is designed for training.

Summary updated

Abstract

Cloud deep learning platforms provide cost-effective deep neural network (DNN) training for customers who lack computation resources. However, cloud systems are often untrustworthy and vulnerable to attackers, leading to growing concerns about model privacy. Recently, researchers have sought to protect data privacy in deep learning by leveraging CPU trusted execution environments (TEEs), which minimize the use of cryptography, but existing works failed to simultaneously utilize the computational resources of GPUs to assist in training and prevent model leakage. This paper presents Tempo, the first cloud-based deep learning system that cooperates with TEE and distributed GPUs for efficient DNN training with model confidentiality preserved. To tackle the challenge of preserving privacy while offloading linear algebraic operations from TEE to GPUs for efficient batch computation, we introduce a customized permutation-based obfuscation algorithm to blind both inputs and model parameters. An optimization mechanism that reduces encryption operations is proposed for faster weight updates during backpropagation to speed up training. We implement Tempo and evaluate it with both training and inference for two prevalent DNNs. Empirical results indicate that Tempo outperforms baselines and offers sufficient privacy protection.
BibTeX
@inproceedings{xu2024tempo,
  author = {Rongwu Xu and Zhixuan Fang},
  title = {{Tempo: Confidentiality Preservation in Cloud-Based Neural Network Training}},
  booktitle = {IJCNN},
  year = {2024},
  url = {https://openreview.net/forum?id=sCQRRMcxHk},
}

Loading related content...

Figure