Quantifying the Performance of Federated Transfer Learning

Qinghe Jing, Weiyan Wang, Junxue Zhang, Han Tian, Kai ChenarXiv 2019 · OpenReview

Abstract

Federated Transfer Learning (FTL) provides a privacy-preserving framework for collaborative machine learning across isolated data silos with differing feature spaces. However, the privacy guarantees of FTL introduce significant performance overheads. This study quantifies these costs using the FATE framework, revealing that FTL can be 17x to 24x slower than traditional distributed machine learning. The primary bottlenecks are identified as inefficient inter-process communication (IPC) and the computational burden of homomorphic encryption.

Introduction

Data silos created by strict privacy regulations like GDPR prevent organizations from pooling data for model training. While standard Federated Learning (FL) addresses this for parties sharing the same feature space, Federated Transfer Learning (FTL) extends collaboration to parties with heterogeneous data. Despite its theoretical appeal, the practical deployment of FTL is hindered by a "performance tax" that has remained largely unquantified until now.

Methodology

The researchers evaluated FTL performance using the Federated AI Technology Enabler (FATE) framework deployed on Google Cloud. They compared FTL against a baseline of standard distributed machine learning (TensorFlow) using a Breast Cancer dataset. The analysis decomposed "Task Completion Time" into CPU execution time (training vs. overhead) and data transfer time, testing both homogeneous and heterogeneous logistic regression scenarios.

Performance Bottlenecks

The study identifies three critical areas where performance degrades:

1. The Efficiency Gap

Only approximately 18% of CPU time is dedicated to actual model training. The remaining 82% is consumed by "extra execution" tasks required to manage the federated environment.

2. Inter-Process Communication (IPC)

The FATE framework's architecture, which mixes Python for machine learning logic and Java for infrastructure, creates a massive IPC bottleneck. Moving data between these environments and managing memory copies accounts for up to 77% of total execution time.

3. Encryption Overhead

FTL utilizes Paillier homomorphic encryption to ensure privacy. This process not only increases CPU load but also significantly inflates data size. A single numerical value can expand to 1024 bits, resulting in 18x more data transfer compared to unencrypted distributed learning.

Comparison of Task Completion, CPU, and Data Transfer Times

Results

The empirical results highlight a stark contrast between FTL and traditional methods:

  • Total Slowdown: FTL was 24x slower for homogeneous models and 17x slower for heterogeneous models compared to standard distributed learning.
  • Network Sensitivity: In cross-region deployments, network latency accounts for up to 34% of total completion time.
  • Encryption Impact: Data encryption alone doubles the training time compared to unencrypted FTL.

Conclusion and Future Work

Current FTL implementations face severe scalability challenges. Increasing hardware resources (vCPU and RAM) yielded diminishing returns, suggesting that software architecture is the primary constraint. Future improvements should focus on optimizing IPC through JVM native memory heaps and UNIX domain sockets, as well as exploring hardware acceleration (FPGAs/ASICs) for homomorphic encryption to make FTL viable for production-scale applications.

Summary updated

Abstract

The scarcity of data and isolated data islands encourage different organizations to share data with each other to train machine learning models. However, there are increasing concerns on the problems of data privacy and security, which urges people to seek a solution like Federated Transfer Learning (FTL) to share training data without violating data privacy. FTL leverages transfer learning techniques to utilize data from different sources for training, while achieving data privacy protection without significant accuracy loss. However, the benefits come with a cost of extra computation and communication consumption, resulting in efficiency problems. In order to efficiently deploy and scale up FTL solutions in practice, we need a deep understanding on how the infrastructure affects the efficiency of FTL. Our paper tries to answer this question by quantitatively measuring a real-world FTL implementation FATE on Google Cloud. According to the results of carefully designed experiments, we verified that the following bottlenecks can be further optimized: 1) Inter-process communication is the major bottleneck; 2) Data encryption adds considerable computation overhead; 3) The Internet networking condition affects the performance a lot when the model is large.
BibTeX
@misc{jing2019quantifying,
  author = {Qinghe Jing and Weiyan Wang and Junxue Zhang and Han Tian and Kai Chen},
  title = {{Quantifying the Performance of Federated Transfer Learning}},
  year = {2019},
  eprint = {1912.12795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1912.12795},
}
Figure