Scalable and Trustworthy Learning in Heterogeneous Networks

Tian LiAAAI 2025 · DOI

Abstract

Modern machine learning increasingly relies on data generated by decentralized networks of devices, such as mobile phones, sensors, and healthcare institutions. This shift from centralized to Federated Learning (FL) introduces significant challenges regarding data heterogeneity and system trustworthiness. This paper synthesizes a series of algorithmic frameworks—including FedProx, q-FFL, Ditto, and TERM—designed to provide stable, fair, and robust optimization in these complex environments. By addressing the core constraints of non-IID data and varying device capabilities, this research establishes a principled foundation for deploying socially responsible and technically sound models at scale.

Introduction

Standard machine learning paradigms typically operate under the assumption that data is centrally located and Identically and Independently Distributed (IID). However, in practical applications, data often remains on local devices to preserve privacy and reduce communication costs. This decentralized approach, known as Federated Learning, must contend with two primary hurdles: heterogeneity and trustworthiness.

This paper provides an overview of research presented at AAAI-25 by Tian Li, which focuses on moving beyond "average-case" performance. The goal is to create optimization objectives that are not only efficient but also rigorously stable and socially responsible when deployed across massive, unreliable networks.

The Challenges of Decentralization

1. Heterogeneity

In large-scale networks, heterogeneity manifests in two forms:

  • System Heterogeneity: Devices possess varying hardware capabilities, battery levels, and network connectivity, leading to inconsistent participation in the training process.
  • Statistical Heterogeneity: Data is distributed across devices in a non-IID fashion. Traditional optimization methods often fail to converge or become highly unstable when local data distributions differ significantly from the global average.

2. Trustworthiness

Training across a massive network introduces risks to the integrity and equity of the model. A global model might achieve high average accuracy while failing catastrophically for specific subgroups (unfairness). Furthermore, decentralized systems are vulnerable to poisoning attacks from malicious actors or simply the presence of extreme outliers that can skew the global model.

Methodological Frameworks

To address these challenges, several key algorithmic contributions have been developed to refine the Federated Learning objective.

FedProx: Stability Under Heterogeneity

FedProx is an optimization framework that introduces a "proximal term" to local objectives. This modification constrains local updates to stay closer to the global model, which provides theoretical convergence guarantees and improved stability under realistic, non-IID data conditions. It effectively handles system heterogeneity by allowing for variable amounts of local work to be performed based on device constraints.

q-Fair FL (q-FFL): Balancing Equity and Utility

Standard FL objectives minimize the average loss, which can lead to models that favor majority groups. The q-FFL framework introduces a reweighted loss objective using a parameter ($q$) to dynamically upweight clients with higher losses. This allows practitioners to tune the tradeoff between average accuracy and fairness, ensuring more uniform performance across the entire network.

Ditto: Personalization as a Defense

Ditto is a multi-task learning framework for FL that enables "personalization." In this approach, each client learns a local model tailored to its own data while remaining regularized toward the global model. This dual-objective approach simultaneously improves fairness (by accounting for local data patterns) and robustness (by limiting the impact of malicious updates on the global model).

Tilted Empirical Risk Minimization (TERM)

TERM provides a generalized objective using exponential "tilting" to control the impact of individual data points. By adjusting the tilting parameter, practitioners can either suppress outliers to improve robustness or prioritize high-loss points to enhance fairness. This single mathematical lens addresses multiple disparate goals in machine learning.

Differentially Private (DP) Adaptive Optimization

Beyond basic Stochastic Gradient Descent (SGD), modern deep learning relies on adaptive optimizers like Adam and AdaGrad. This research introduces new algorithms that leverage gradient structures to improve the privacy-utility tradeoff, providing provable privacy guarantees without sacrificing the performance benefits of adaptive optimization.

Key Insights and Results

The synthesis of these methods reveals that heterogeneity is not merely a technical nuisance but the core constraint linking accuracy, fairness, and robustness. Key findings include:

  • Theoretical Stability: FedProx provides the first rigorous convergence guarantees for heterogeneous FL, outperforming standard Federated Averaging (FedAvg) in stability.
  • Scalable Fairness: q-FFL offers a tunable and scalable method for implementing fairness in production environments.
  • Unified Robustness: The TERM objective provably decreases the influence of outliers, leading to better generalization across diverse data distributions.
  • Privacy Integration: Adaptive optimization can be made differentially private while maintaining high accuracy, bridging the gap between theoretical privacy and practical deep learning needs.

Conclusion and Future Work

While significant progress has been made in stabilizing Federated Learning, simultaneously satisfying accuracy, fairness, and robustness remains exceptionally difficult under extreme heterogeneity. Future research directions include refining optimization for even more massive, unstable networks, exploring principled data-sharing mechanisms that do not compromise privacy, and further narrowing the gap between theoretical differential privacy and practical application requirements.

Summary updated

BibTeX
@inproceedings{li2025scalable,
  author = {Tian Li},
  title = {{Scalable and Trustworthy Learning in Heterogeneous Networks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35110},
}
Figure