Privacy-Preserving Vertical Federated Learning

Runhua Xu, Nathalie Baracaldo, Yi Zhou, Annie Abay, Ali AnwarFederated Learning 2022 · OpenReview

TL;DR

This paper introduces FedV, a novel framework for privacy-preserving federated learning (FL) in vertically partitioned datasets. FedV significantly reduces training time and data transfer by eliminating peer-to-peer communication and enabling the training of various machine learning models without approximations, unlike prior methods.

What Problem Does This Paper Address?

Federated learning (FL) allows multiple parties to collaboratively train a machine learning (ML) model without sharing their raw data. While much research focuses on horizontal FL (where each party has the same features but different data samples), many real-world scenarios involve vertical FL (VFL). In VFL, parties have complementary feature sets, and often only one party possesses the labels. This vertical partitioning poses a significant challenge for privacy-preserving ML because a complete dataset is only formed when data from multiple parties is combined, and privacy regulations restrict data sharing. Existing VFL solutions are often limited to specific models (e.g., linear regression), require extensive peer-to-peer communication, and may rely on approximations that reduce model accuracy. This paper aims to bridge this gap by proposing FedV, a generic and efficient VFL framework that addresses these limitations.

Methods

FedV's core innovation lies in its use of functional encryption (FE) schemes to enable secure gradient computation without direct peer-to-peer communication between parties. The framework operates in two main phases:

  1. Private Entity Resolution (PER): This phase aligns data records across parties. FedV uses cryptographic long-term keys (CLK) and the Dice coefficient for matching, followed by an aggregator shuffling datasets based on a permutation vector. This ensures records are matched privately.
  2. Federated Vertical Secure Gradient Descent (FedV-SecGrad): This is the heart of FedV. It employs a two-phased secure aggregation (2Phased-SA) approach:
  3. Feature-dimension Secure Aggregation: Parties encrypt their "partial models" (features combined with model weights, and for the active party, labels) using multi-input functional encryption (MIFE). The aggregator then uses a functional decryption key (obtained from a Trusted Third Party - TPA, or in a TPA-free manner) to compute an aggregated value representing the loss component for each feature.
  4. Sample-dimension Secure Aggregation: Parties encrypt their data samples using single-input functional encryption (SIFE). The aggregator uses the results from the feature-dimension SA and a functional decryption key to compute the final batch gradient.

This approach avoids peer-to-peer communication, relying solely on communication between parties and the aggregator. FedV also incorporates an Inference Prevention Module (IPM) and random batch selection to thwart potential inference attacks by the aggregator.

Datasets

The paper evaluates FedV on several publicly available datasets from the UCI Machine Learning Repository:

  • Phishing
  • Ionosphere
  • Statlog
  • OptDigits
  • MNIST

These datasets were partitioned vertically for the experiments.

Key Insights and Ideas

  • Functional Encryption for VFL: The paper demonstrates that functional encryption, particularly MIFE and SIFE, can be effectively leveraged to build a generic VFL framework, moving beyond model-specific or limited cryptographic solutions.
  • Eliminating Peer-to-Peer Communication: By shifting computation to the aggregator via FE, FedV drastically reduces communication overhead, a major bottleneck in existing VFL methods.
  • Generic Model Support: FedV's design, based on secure gradient computation, allows it to support a wide range of ML models (linear regression, logistic regression, SVMs) without requiring approximations like Taylor series expansion, leading to potentially higher accuracy.
  • Dynamic Participation: The framework is designed to handle parties that may drop out and rejoin during training, enhancing its robustness in real-world, less stable environments.
  • Privacy Guarantees: FedV provides strong privacy guarantees against inference attacks by the aggregator, thanks to its secure aggregation protocols and the IPM.

Key Results

  • Training Time Reduction: FedV achieved a reduction of 10%-70% in training time compared to the state-of-the-art Hardy et al. [26] approach for logistic regression.
  • Data Transfer Reduction: FedV reduced data transfer by 80%-90% compared to Hardy et al. [26].
  • Model Accuracy: FedV achieved comparable test accuracy to centralized baselines and Hardy et al. [26] for logistic regression. For other models like linear SVM, FedV achieved comparable accuracy to centralized models.
  • Scalability: FedV demonstrated good performance with an increasing number of parties, showing a linear relationship between training time and the number of parties.
  • Model Applicability: FedV successfully trained models that baseline approaches (like Hardy) could not, particularly for non-linear models without approximations.

Limitations and Future Work

  • Trusted Third Party (TPA): While FedV can be TPA-free with certain FE schemes, the described implementation relies on a TPA for functional decryption key generation. The paper notes that TPA-free FE schemes are an active research area.
  • Integer-based Cryptography: FedV's reliance on integer-based cryptography means it may lose some precision compared to floating-point computations, potentially leading to minor accuracy differences.
  • Future Work: The paper suggests exploring TPA-free FE schemes more deeply and investigating FedV's applicability to more complex ML models and larger-scale distributed systems. The authors also mention the potential for further optimization of the secure aggregation protocols.

Summary updated

Abstract

Many federated learning (FL) proposals follow the structure of horizontal FL, where each party has all the necessary information to train a model available to them. However, in important real-world FL scenarios, not all parties have access to the same information, and not all have what is required to train a machine learning model. In what is known as vertical scenarios, multiple parties provide disjoint sets of information that, when brought together, can create a full feature set with labels, which can be used for training. Legislation, practical considerations, and privacy requirements inhibit moving all data to a single place or freely sharing among parties. Horizontal FL techniques cannot be applied to vertical settings. This chapter discusses the use cases and challenges of vertical FL. It introduces the most important approaches for vertical FL and describes in detail FedV, an efficient solution to perform secure gradient computation for popular ML models. FedV is designed to overcome some of the pitfalls inherent to applying existing state-of-the art techniques. Using FedV substantially reduces training time and the amount of data transfer and enables the use of vertical FL in more real-world use cases.
BibTeX
@inproceedings{xu2022privacy,
  author = {Runhua Xu and Nathalie Baracaldo and Yi Zhou and Annie Abay and Ali Anwar},
  title = {{Privacy-Preserving Vertical Federated Learning}},
  booktitle = {Federated Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=UpXACw1N5R},
}
Figure