Parameter-free HE-friendly Logistic Regression

Junyoung Byun, Woojin Lee, Jaewook LeeNeurIPS 2021 · OpenReview

TL;DR

This paper introduces a novel framework for training logistic regression models on encrypted data without the typical headaches of hyperparameter tuning or sigmoid approximations. By transforming the classification task into a ridge regression problem for "logits" and leveraging a small amount of unencrypted data, the authors achieve superior accuracy while slashing training latency by approximately 68%.

The Challenge of Encrypted Logistic Regression

Machine Learning on Encrypted Data (MLED) using Homomorphic Encryption (HE) is the "holy grail" of privacy-preserving AI, allowing models to be trained without ever exposing the raw data. However, training a standard Logistic Regression (LR) model under HE is notoriously difficult for two primary reasons:

  1. The Sigmoid Problem: HE only supports polynomial operations (addition and multiplication). The LR sigmoid function is non-polynomial and must be approximated by a low-degree polynomial. If the input data falls outside the approximation's narrow range, the model's accuracy collapses.
  2. The Hyperparameter Trap: Standard training uses Gradient Descent (GD), which requires tuning the learning rate and the number of iterations. In an encrypted environment, traditional cross-validation is impossible because intermediate loss values are hidden. Picking the wrong parameters can lead to a model that fails to converge.
Challenges of Encrypted LR: Hyperparameter instability and sigmoid approximation errors

As illustrated above, standard gradient descent is highly sensitive to the learning rate ($\eta$), with improper values causing the loss to spike erratically. Furthermore, polynomial approximations of the sigmoid function diverge wildly outside their specified ranges, corrupting the model's predictions.

A Parameter-Free Framework

The authors propose a "Parameter-free" framework that bypasses these issues entirely. Instead of forcing standard logistic regression to work within the constraints of HE, they reframe the problem.

Proposed Framework Pipeline

The methodology relies on three key steps:

  • Teacher Modeling: The framework assumes the modeler has access to a small, unencrypted dataset ($D_1$). A simple "teacher" model ($f_s$) is trained on this plaintext data to extract predicted probabilities (logits) for labels.
  • Mean Matching: Because the unencrypted data ($D_1$) and the encrypted target data ($D_2$) might come from different distributions, a regularization step is introduced to shift the logits. This transformation ensures the teacher's knowledge transfers effectively to the encrypted domain.
  • Ridge Regression for Logits: Instead of training a standard LR on encrypted data, the authors solve a Ridge Regression problem where the target variable is the shifted logit. Ridge regression has a closed-form solution. By using the Sherman-Woodbury matrix identity, the authors separate operations on encrypted private variables from non-encrypted ones. This turns a massive matrix inversion into a much smaller, manageable calculation, eliminating the need for iterative Gradient Descent and sigmoid approximations.

Experimental Validation

The researchers evaluated their method on five widely used classification datasets from the UCI repository: Adult (Income), Bank (Marketing), Cancer (Breast Cancer detection), Diabetes, and Credit. In these tests, specific high-impact or socially biased features were treated as "private" (encrypted) variables, while others remained plaintext.

Key Results

  • Speed: The proposed method reduced training latency by ~68% compared to the previous state-of-the-art encrypted LR model (LRHE).
  • Accuracy: The parameter-free approach consistently outperformed the baseline across all datasets. For example, on the Cancer dataset, accuracy improved from 86.9% to 90.5%.
  • Stability: Unlike previous models where performance swung wildly based on the chosen learning rate, this method remained perfectly stable because it relies on a closed-form solution rather than gradient-based optimization.
  • Distribution Correction: In ablation studies where the teacher and encrypted data were intentionally different, the "Mean Matching" step successfully recovered the classification performance that would have otherwise been lost.

Key Insights

  • Logit Space is Safer: By moving the problem from the probability space (0 to 1) to the logit space ($-\infty$ to $+\infty$), the model completely avoids the catastrophic errors introduced by polynomial approximations of the sigmoid function.
  • Partial Encryption is a Feature, Not a Bug: The paper leans into the reality that not all data is equally sensitive. By only encrypting "private variables," they achieve a massive "quadratic reduction" in computation cost.
  • Closed-Form > Iterative: Moving from Gradient Descent to a closed-form solution isn't just faster; it removes the "blind" hyperparameter tuning that makes MLED so risky and impractical in production.

Limitations and Future Directions

While highly effective, the core framework is designed for linear relationships. Although the authors mention a kernel-based extension for non-linear data, the primary focus remains on linear models. Additionally, the security model assumes participants are "honest-but-curious" and does not currently protect against malicious actors intentionally providing corrupted data. Finally, the method relies on having at least some unencrypted data to train the teacher model; scenarios with zero plaintext data would require a different initialization strategy. Future work will likely focus on strengthening the security model and exploring more complex non-linear architectures within this logit-regression framework.

Summary updated

Abstract

Privacy in machine learning has been widely recognized as an essential ethical and legal issue, because the data used for machine learning may contain sensitive information. Homomorphic encryption has recently attracted attention as a key solution to preserve privacy in machine learning applications. However, current approaches on the training of encrypted machine learning have relied heavily on hyperparameter selection, which should be avoided owing to the extreme difficulty of conducting validation on encrypted data. In this study, we propose an effective privacy-preserving logistic regression method that is free from the approximation of the sigmoid function and hyperparameter selection. In our framework, a logistic regression model can be transformed into the corresponding ridge regression for the logit function. We provide a theoretical background for our framework by suggesting a new generalization error bound on the encrypted data. Experiments on various real-world data show that our framework achieves better classification results while reducing latency by $\sim68\%$, compared to the previous models.
BibTeX
@inproceedings{byun2021parameter,
  author = {Junyoung Byun and Woojin Lee and Jaewook Lee},
  title = {{Parameter-free HE-friendly Logistic Regression}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=0DBYkHfkZlk},
}
Figure