A Systematic Literature Review on Federated Learning: From A Model Quality Perspective

Yi Liu, Li Zhang, Ning Ge, Guanghao LiarXiv 2020 · arXiv

Introduction

The rapid advancement of machine learning is increasingly constrained by stringent privacy regulations, such as the General Data Protection Regulation (GDPR). These legal frameworks have effectively created "data islands"—isolated silos of information that cannot be aggregated into centralized repositories for traditional training. Federated Learning (FL) has emerged as a transformative paradigm to address this, enabling models to be trained locally on distributed devices while sharing only encrypted updates with a central coordinator.

Despite its promise, a significant barrier to FL adoption is the perceived trade-off between privacy and performance. Practitioners often fear that decentralized training on noisy, heterogeneous data will result in inferior model quality. This systematic literature review (SLR) analyzes 147 core articles published between 2016 and early 2020 to evaluate the current state of FL model quality and provide a structured path for optimization.

Methodology and the FL Application Framework

The authors conducted a rigorous three-stage selection process, filtering an initial pool of 1,140 papers down to 147 high-impact studies from major databases including IEEE, ACM, and Google Scholar. A primary contribution of this work is the synthesis of these findings into a comprehensive FL Application Framework.

The proposed Federated Learning Application Framework illustrating the interaction between server, clients, privacy mechanisms, and incentive structures.

This framework guides practitioners through six critical phases of development:

  1. Application Scene Analysis: Balancing requirements against computational and communication costs.
  2. Data Characteristic Analysis: Evaluating the intersection of features and samples across participants.
  3. Federated Setting Selection: Determining the appropriate architecture (Horizontal, Vertical, or Transfer).
  4. Algorithm Selection: Choosing between client-side and server-side optimization strategies.
  5. Privacy Protection Design: Managing the trade-offs between encryption strength and model accuracy.
  6. Incentive Mechanism Design: Implementing rewards to ensure the participation of high-quality data contributors.

Categorizing Federated Learning Settings

A key factor in model quality is selecting the correct FL setting based on how data is distributed across the network. The review categorizes these into three primary types based on the overlap of sample IDs and feature spaces:

Classification of Federated Learning based on the distribution of samples and features: Horizontal FL, Vertical FL, and Federated Transfer Learning.
  • Horizontal Federated Learning: Applied when datasets share the same feature space but differ in samples (e.g., regional hospitals with similar records but different patients). This is currently the most researched area.
  • Vertical Federated Learning: Used when datasets share the same sample ID space but differ in feature space (e.g., a bank and an e-commerce platform sharing the same customer base).
  • Federated Transfer Learning: Employed when datasets differ in both samples and features, requiring transfer learning techniques to bridge the gap.

Key Insights: The "5% Rule" and Beyond

The review reveals several critical insights regarding the performance of federated systems:

  • The 5% Performance Gap: In approximately 92% of the reviewed studies, FL achieved model performance within 1–5% of its centralized counterparts. This suggests that FL is a viable alternative for high-stakes applications.
  • The Encryption Tax: There is an inherent tension between privacy and utility. Stronger privacy measures, such as high-noise Differential Privacy, often lead to a measurable decline in accuracy—a challenge that remains a focal point for ongoing research.
  • Incentives as a Quality Driver: Model quality is not purely a technical challenge; it is also an economic one. Without robust incentive mechanisms (e.g., Stackelberg games or auctions), participants with the most valuable data may opt out, leading to biased global models.

The velocity of FL research is accelerating rapidly. The number of publications in the first quarter of 2020 alone nearly matched the total output of 2018. Geographically, China (39%) and the USA (27%) lead the field. The most prominent application domains include Edge Computing/IoT (33%) and Healthcare (29%), where data sensitivity is highest.

While FedAvg remains the foundational aggregation algorithm, newer variants like FedProx and FedMA have shown superior stability in "Non-IID" (heterogeneous) data environments. Furthermore, "Federated Personalization" is emerging as a way to allow clients to maintain local model components, sometimes resulting in local performance that exceeds centralized benchmarks.

Challenges and Future Directions

Despite significant progress, several hurdles remain:

  1. Non-IID Data: Training on non-identically distributed data continues to cause convergence issues for standard algorithms.
  2. Vertical FL Maturity: There is a notable lack of open-source frameworks and empirical studies for Vertical FL compared to Horizontal FL.
  3. Communication Overhead: The cost of transmitting large model updates remains a bottleneck, particularly for resource-constrained mobile and IoT devices.
  4. Empirical Incentive Models: Most incentive structures remain theoretical; real-world validation of how users respond to different reward systems is needed.

In conclusion, while privacy protection does impose certain constraints, the performance hit is often negligible when the right framework and algorithms are applied. As FL matures, the focus is shifting from basic feasibility to the nuanced optimization of model quality across diverse, real-world ecosystems.

Summary updated

BibTeX
@misc{liu2020systematic,
  author = {Yi Liu and Li Zhang and Ning Ge and Guanghao Li},
  title = {{A Systematic Literature Review on Federated Learning: From A Model Quality Perspective}},
  year = {2020},
  eprint = {2012.01973},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.01973},
}
Figure