A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification

Veneta Haralampieva, Daniel Rueckert, Jonathan Passerat-Palmbach

Introduction

As Deep Learning becomes ubiquitous, the "Machine Learning-as-a-Service" (MLaaS) model has created a significant privacy paradox. To obtain a prediction, a user must typically send raw, sensitive data—such as medical records or financial statements—to a cloud provider. Conversely, the provider seeks to protect their model weights, which often represent expensive intellectual property.

While several technologies exist to address this—Secure Multi-Party Computation (SMPC), Homomorphic Encryption (HE), and Trusted Execution Environments (TEE)—the landscape remains fragmented. Most research focuses on theoretical protocols rather than the practical performance of these libraries within the standard "data science stack" (Python, PyTorch, and TensorFlow). This study fills that gap by systematically benchmarking these tools on standardized hardware and datasets.

Methodology

The evaluation focused on four frameworks representing three distinct privacy paradigms:

  1. SMPC (Secure Multi-Party Computation): Represented by PySyft and CrypTen. These frameworks split data into "secret shares" distributed across multiple parties. No single party can view the complete data, yet they can collectively compute results.
  2. TEE (Trusted Execution Environments): Represented by TF-Trusted (utilizing Intel SGX). This approach creates a secure "enclave" within the CPU hardware where data is decrypted and processed in isolation from the rest of the system.
  3. HE (Homomorphic Encryption): Represented by HE-Transformer. This allows mathematical operations to be performed directly on encrypted data without ever requiring decryption.

The study involved training "plaintext" models first, then porting them to these frameworks to measure the overhead of "private inference" (making predictions on encrypted data).

Experimental Setup

The study utilized two primary datasets to test varying levels of complexity:

  • MNIST: A classic benchmark consisting of 70,000 grayscale images of handwritten digits (0-9).
  • Malaria Cell Images: A more complex medical imaging dataset containing 27,558 images categorized as "infected" or "uninfected." This dataset was used to evaluate how frameworks handle deeper, more realistic architectures (Model Z).

Key Insights

Accuracy Preservation

A major takeaway is that all four frameworks perfectly preserved the accuracy of the original plaintext models. This demonstrates that the "privacy tax" is paid in computational time and network bandwidth, rather than in the quality of the predictions.

The ReLU Bottleneck

In SMPC, non-linear functions like ReLU are computationally expensive. The authors found that CrypTen significantly outperformed PySyft because CrypTen utilizes Garbled Circuits for ReLUs, which requires substantially less communication than the additive sharing method used in PySyft.

Communication and Latency

HE-Transformer exhibited staggering communication overhead. For the Malaria dataset (Model Z), it required transferring approximately 56 GB of data for a single image prediction. This highlights that while HE is mathematically elegant, it is currently impractical for deep convolutional networks in standard network environments. In contrast, TF-Trusted (the TEE approach) provided a very consistent runtime (approximately 0.14s) regardless of model complexity, as it essentially runs plaintext code inside a secure hardware "black box."

Performance Results

The benchmarks revealed a clear hierarchy of performance across the tested models (X, Y, and Z).

Performance comparison of PPML frameworks across different models
  • Speed: CrypTen and TF-Trusted emerged as the most efficient. On MNIST (Model X), CrypTen achieved inference in ~0.043s. On the more complex Malaria dataset (Model Z), TF-Trusted took 0.14s, while CrypTen took 0.25s.
  • Efficiency: PySyft was notably slower on complex models, requiring over 8 seconds per image for Model Z.
  • The HE Gap: HE-Transformer was the extreme outlier, taking nearly 549 seconds (over 9 minutes) for a single Malaria cell classification, compared to less than a second for the other methods.

Limitations and Future Work

The authors identify several hurdles for practical deployment:

  • Hardware Constraints: TEEs like Intel SGX have limited "protected memory," making it difficult to run massive state-of-the-art models without specialized engineering.
  • HE Optimization: Current HE implementations are often too generic. Bespoke protocols or improved automatic toolchains are necessary to make HE competitive for deep learning.
  • Real-world Networking: These tests were conducted in a LAN setting. In real-world scenarios, the high communication overhead of SMPC and HE would be even more pronounced over standard internet connections.

Future research directions include investigating batched predictions—where HE might show better amortized performance—and the impact of model quantization on private inference efficiency.

Summary updated

BibTeX
@inproceedings{haralampieva2020systematic,
  author = {Veneta Haralampieva and Daniel Rueckert and Jonathan Passerat-Palmbach},
  title = {{A Systematic Comparison of Encrypted Machine Learning Solutions for Image Classification}},
  booktitle = {PPMLP@CCS},
  year = {2020},
  doi = {10.1145/3411501.3419432},
}
Figure