A Systematic Literature Review on Federated Machine Learning: From A Software Engineering Perspective

Sin Kit Lo, Qinghua Lu, Chen Wang, Hye-Young Paik, Liming Zhu

Introduction

Federated Machine Learning (FL) has emerged as a transformative paradigm for training machine learning models on decentralized data. By allowing clients to train models locally and sharing only model updates with a central aggregator, FL addresses critical concerns regarding data privacy and bandwidth. However, while the mathematical foundations of FL—such as gradient optimization and loss functions—have seen significant advancement, the practical challenges of building and maintaining FL as a distributed software system remain under-explored.

This paper presents a systematic literature review (SLR) of 231 primary studies, uniquely analyzing the field through the lens of Software Engineering (SE). By mapping current research to the Software Development Life Cycle (SDLC), we identify the state-of-the-art and highlight the gaps between theoretical research and production-ready systems.

Methodology

To ensure a comprehensive and unbiased review, we followed the established Kitchenham guidelines for systematic reviews. Our process involved an initial search across major digital libraries, including ACM, IEEE, ScienceDirect, and arXiv, covering the period from 2016 to early 2020.

The systematic literature review process, including paper screening and snowballing.

As shown in the figure above, the methodology included rigorous paper screening based on inclusion and exclusion criteria, followed by a bi-directional snowballing process to ensure no significant studies were omitted. This resulted in a final set of 231 primary studies for data synthesis.

The Software Engineering Perspective

The core contribution of this review is the categorization of FL research into the standard phases of the software engineering lifecycle. We formulated specific research questions (RQs) to investigate how the community addresses background understanding, requirement analysis, architecture design, and implementation/evaluation.

Mapping research questions to the software development lifecycle phases.

1. Background and Requirements

Our analysis reveals that privacy is the primary driver for FL adoption, cited by 62% of the studies. From a requirement analysis standpoint, there is a heavy focus on non-functional requirements like communication efficiency (23%). However, other critical SE requirements, such as system auditability and user incentives ("client motivatability"), are significantly under-represented.

2. Architecture and Design

FL systems are fundamentally distributed. While the goal is decentralization, we observed a "centralization paradox": most current architectures still rely on a central server for orchestration, creating potential single points of failure. We identified a taxonomy of components, distinguishing between mandatory elements (like local model trainers) and optional but critical enhancements (like anomaly detectors and model compressors).

3. Implementation and Evaluation

A significant "lab-to-production" gap exists. Approximately 79% of the literature is classified as "validation research" based on simulations, while only 6% represents "evaluation research" involving real-world case studies. Furthermore, nearly 70% of research focuses exclusively on the training phase, neglecting deployment, monitoring, and maintenance.

Data Landscape and Benchmarks

The review catalogs the types of data used in FL research. Image data is the most dominant (49%), with the MNIST dataset serving as the most frequent benchmark for proof-of-concept. Structured data (21%) and text data (14%) follow. Notably, there is a lack of research utilizing IoT time-series data or complex graph data, despite the frequent marketing of FL for edge computing and IoT scenarios.

Key Insights

  • Non-IID Challenges: Handling Independent and Identically Distributed (Non-IID) data remains a top-tier challenge. When every user's data distribution differs, model performance often degrades, and standardized SE solutions for this are still immature.
  • The Training Bias: The disproportionate focus on model training means that the "unsexy" parts of the lifecycle—such as data cleaning at the edge and versioning—are largely ignored.
  • Emerging Trends: There is a nascent but growing trend toward using blockchains for decentralized data governance to resolve the centralization paradox.

Future Directions

To advance FL from a research topic to a robust engineering discipline, future work must address the full lifecycle: moving beyond training to include edge-side data preparation, model deployment, and long-term performance monitoring. Additionally, researchers should explore semi-supervised FL for unlabeled client data and investigate how FL systems can comply with regulatory mandates like the "Right to be Forgotten" under GDPR.

By bridging the gap between ML algorithms and software engineering practices, we can build more reliable, scalable, and privacy-preserving federated systems.

Summary updated

Abstract

Federated learning is an emerging machine learning paradigm where clients train models locally and formulate a global model based on the local model updates. To identify the state-of-the-art in federated learning and explore how to develop federated learning systems, we perform a systematic literature review from a software engineering perspective, based on 231 primary studies. Our data synthesis covers the lifecycle of federated learning system development that includes background understanding, requirement analysis, architecture design, implementation, and evaluation. We highlight and summarise the findings from the results, and identify future trends to encourage researchers to advance their current work.
BibTeX
@article{lo2022systematic,
  author = {Sin Kit Lo and Qinghua Lu and Chen Wang and Hye-Young Paik and Liming Zhu},
  title = {{A Systematic Literature Review on Federated Machine Learning: From A Software Engineering Perspective}},
  journal = {ACM Computing Surveys},
  year = {2022},
  url = {https://openreview.net/forum?id=nh95E4LEDyH},
}
Figure