Residue-based Label Protection Mechanisms in Vertical Logistic Regression

Juntao Tan, Lan Zhang, Yang Liu, Anran Li, Ye Wu

Introduction

Vertical Federated Learning (VFL) has emerged as a powerful paradigm for collaborative machine learning, allowing organizations with different feature sets for the same group of users to train models without sharing raw data. A typical scenario involves an "active" party, who holds the labels (e.g., loan default status), and a "passive" party, who contributes additional features (e.g., e-commerce transaction history). While VFL is designed to preserve privacy by only exchanging intermediate gradients rather than raw data, recent research has exposed a critical vulnerability: the passive party can often infer the active party's private labels through gradient analysis.

This paper investigates this leakage and proposes three residue-based protection mechanisms designed to safeguard labels while maintaining model utility. These range from computationally efficient noise-based methods to a sophisticated hybrid approach that achieves lossless training.

The Vulnerability: Label Leakage via Residues

The core privacy flaw in standard VFL protocols lies in the mathematical structure of the training gradients. In logistic regression, the gradient sent to the passive party is a product of their local features and the "residue"—the difference between the model's prediction and the ground truth label ($y - \hat{y}$).

Researchers discovered that if a passive party (Alice) possesses a sufficient number of features ($d^A$) relative to the mini-batch size ($|Batch|$), she can treat the received gradients as a system of linear equations. When $d^A > |Batch|$, the system becomes overdetermined, allowing Alice to solve for the exact residues. Once the residues are known, the binary labels can be reconstructed with high accuracy. This "Equation Solving" attack is particularly potent in real-world scenarios where high-dimensional feature sets are common.

Proposed Defense Mechanisms

To mitigate this risk, the authors introduce three Residue-based Label Protection Mechanisms:

1. Additive Noise Mechanism ($M_{add}$)

In this approach, the label holder (Bob) adds Laplace noise to the residues before transmitting them to Alice. This method satisfies $\epsilon$-Local Differential Privacy (LDP). While $M_{add}$ is computationally fast and easy to implement, it introduces a fundamental trade-off: increasing privacy (lower $\epsilon$) injects more noise, which can degrade the final model's accuracy.

2. Multiplicative Noise Mechanism ($M_{mult}$)

Instead of addition, Bob multiplies the residues by Laplace noise. This mechanism also adheres to $\epsilon$-LDP. To manage the sensitivity of the residues and ensure privacy guarantees, the authors employ two clipping steps. Like the additive approach, $M_{mult}$ is efficient but can impact model performance depending on the privacy budget.

3. Hybrid Mechanism ($M_{hybrid}$)

The most advanced solution proposed is a hybrid approach that combines Random Response (an LDP technique) with Additive Homomorphic Encryption (HE).

  • Random Response: This is used to obscure which specific samples are included in a mini-batch by adding "dummy" samples. This prevents the attacker from correctly setting up the linear equations needed for the attack.
  • Homomorphic Encryption: The residues are encrypted such that Alice can perform the necessary gradient calculations without ever seeing the underlying values.
  • Lossless Training: By setting the residues of dummy samples to zero within the encrypted space, the mechanism ensures the final aggregated gradient remains mathematically identical to the true gradient. This allows for "lossless" training, where the private model achieves the exact same accuracy as a non-private, centralized model.

Experimental Evaluation

The mechanisms were evaluated across four diverse binary classification datasets: breast-cancer, sklearn-digits, census-income, and give-me-some-credit.

Key Findings

  • Accuracy and Privacy: $M_{hybrid}$ consistently achieved zero accuracy degradation across all datasets, successfully neutralizing the label inference attack without compromising the model. For the noise-based methods ($M_{add}$ and $M_{mult}$), high privacy settings (e.g., $\epsilon = 0.01$) led to significant accuracy drops, though they remained effective at more relaxed privacy levels ($\epsilon = 10$).
  • Computational Overhead: While encryption-based methods are traditionally slow, $M_{hybrid}$ demonstrated a manageable overhead, with training times only 1.3x to 1.8x longer than the standard, unprotected VFL protocol.
  • Regularization Effects: Interestingly, on certain datasets like sklearn-digits, the introduction of small amounts of noise via $M_{add}$ or $M_{mult}$ actually improved the model's Area Under the Curve (AUC), suggesting that the privacy-preserving noise can occasionally act as a form of regularization.

Conclusion and Future Work

This research demonstrates that while VFL faces significant label leakage risks, effective defenses are available. The $M_{hybrid}$ mechanism, in particular, offers a compelling path forward by providing robust privacy without sacrificing model utility.

Future research directions include automating the tuning of clipping bounds for multiplicative noise and optimizing homomorphic encryption to ensure these defenses scale to massive deep learning architectures. Additionally, while these mechanisms protect against "honest-but-curious" participants, further work is needed to secure VFL against malicious actors who might actively attempt to subvert the protocol.

Summary updated

BibTeX
@inproceedings{tan2022residue,
  author = {Juntao Tan and Lan Zhang and Yang Liu and Anran Li and Ye Wu},
  title = {{Residue-based Label Protection Mechanisms in Vertical Logistic Regression}},
  booktitle = {BigCom},
  year = {2022},
  url = {https://openreview.net/forum?id=xNbXDnnwV6Q},
}
Figure