GAS: Generative Activation-Aided Asynchronous Split Federated Learning

Jiarong Yang, Yuan Liu

TL;DR

This paper introduces GAS, a novel asynchronous Split Federated Learning (SFL) framework that tackles the performance degradation caused by client stragglers. By employing an activation buffer, a model buffer, and a unique generative activation mechanism, GAS ensures more balanced and accurate server-side model updates, leading to improved convergence and performance.

What Problem Does This Paper Address?

Federated Learning (FL) and Split Learning (SL) are powerful techniques for distributed machine learning, but they face challenges in real-world scenarios. Split Federated Learning (SFL) combines the strengths of both, splitting a model between clients and a server. Clients train the initial layers and send intermediate activations to the server, which completes the model. However, traditional SFL assumes synchronous communication, meaning the server waits for all clients to send their updates before proceeding. This is problematic because clients have varying computational and communication capabilities, leading to "stragglers" (slow clients) that significantly delay training and degrade performance. Previous attempts to address this in SFL have limitations, either being constrained by model structure or still requiring the server to wait for stragglers. This paper aims to solve the problem of asynchronous updates in SFL by developing a framework that can handle varying client speeds without sacrificing performance.

Methods

The core innovation of GAS lies in its asynchronous framework and the "Generative Activation-Aided Updates" mechanism.

  • Asynchronous Framework: GAS introduces an activation buffer and a model buffer on the server. The activation buffer stores incoming activations from clients, and the model buffer stores client-side models. When these buffers reach a certain capacity, the server processes them, allowing for continuous updates without waiting for all clients.
  • Generative Activation-Aided Updates: This is the key to mitigating bias introduced by asynchronous updates.
  • Dynamic Weighted Update: The server maintains a distribution (Gaussian) for activations of each label. This distribution is dynamically updated using incoming activations, with newer activations weighted more heavily.
  • Generation and Concatenation: When the activation buffer is full, the server generates synthetic activations from these distributions. These generated activations are then concatenated with the actual received activations before updating the server-side model. This process helps to balance the dataset representation, compensating for the potential bias introduced by resource-rich clients sending more frequent updates.
  • Theoretical Analysis: The paper provides theoretical guarantees for GAS, showing a tighter convergence bound for the server-side model due to reduced gradient dissimilarity. It also suggests that a decaying learning rate can further mitigate the impact of stragglers over time.

Datasets

The paper evaluates GAS on the following datasets:

  • CIFAR-10
  • CINIC-10
  • Fashion-MNIST

These datasets are standard benchmarks for image classification tasks. The authors also simulate data heterogeneity using both shard-based and Dirichlet distribution-based methods, which are common techniques to mimic real-world data imbalances.

Key Insights and Ideas

  • Generative Activations as a Bias Corrector: The most significant insight is the use of generated activations to counteract the bias introduced by asynchronous updates from clients with varying capabilities. This is a clever way to "fill in the gaps" and ensure a more representative update for the server-side model.
  • Buffer Management for Asynchrony: The introduction of activation and model buffers is a practical approach to handle the inherent asynchrony in distributed learning, enabling continuous progress without strict synchronization.
  • Theoretical Support for Practical Improvements: The theoretical analysis provides a solid foundation for the observed performance gains, demonstrating how GAS achieves a tighter convergence bound by reducing gradient dissimilarity.
  • Dynamic Weighting for Temporal Relevance: The weighted update of activation distributions, favoring newer data, acknowledges that more recent information is generally more relevant in a dynamic training process.

Key Results

  • Improved Accuracy: GAS consistently outperforms baseline asynchronous FL algorithms (FedBuff, CA2FL) and synchronous SFL algorithms (MiniBatch-SFL, S²FL) across various datasets and heterogeneity settings, achieving higher test accuracy. For example, on CIFAR-10 with shard=2, GAS achieved 82.78% accuracy, outperforming FedBuff (69.04%) and CA2FL (79.57%).
  • Faster Convergence: GAS demonstrates significantly faster convergence, especially in real-world communication environments. For instance, GAS achieved 60% accuracy in 30 minutes with fewer local iterations, while CA2FL required 60 minutes for the same accuracy with more local iterations.
  • Reduced Gradient Dissimilarity: Experimental validation confirms that generative activations effectively reduce gradient dissimilarity, supporting the theoretical claims.

Limitations and Future Work

  • Privacy Concerns: The paper acknowledges that, like other SL and SFL algorithms, GAS requires the transmission of labels and activations, which poses a privacy risk.
  • Future Work: The authors suggest incorporating privacy-preserving mechanisms into GAS to enhance data security and broaden its applicability. This is a crucial step for real-world deployment.

Summary updated

Abstract

Split Federated Learning (SFL) splits and collaboratively trains a shared model between clients and server, where clients transmit activations and client-side models to server for updates. Recent SFL studies assume synchronous transmission of activations and client-side models from clients to server. However, due to significant variations in computational and communication capabilities among clients, activations and client-side models arrive at server asynchronously. The delay caused by asynchrony significantly degrades the performance of SFL. To address this issue, we consider an asynchronous SFL framework, where an activation buffer and a model buffer are embedded on the server to manage the asynchronously transmitted activations and client-side models, respectively. Furthermore, as asynchronous activation transmissions cause the buffer to frequently receive activations from resource-rich clients, leading to biased updates of the server-side model, we propose Generative activations-aided Asynchronous SFL (GAS). In GAS, the server maintains an activation distribution for each label based on received activations and generates activations from these distributions according to the degree of bias. These generative activations are then used to assist in updating the server-side model, ensuring more accurate updates. We derive a tighter convergence bound, and our experiments demonstrate the effectiveness of the proposed method. The code is available at https://github.com/eejiarong/GAS.
BibTeX
@inproceedings{yang2025gas,
  author = {Jiarong Yang and Yuan Liu},
  title = {{GAS: Generative Activation-Aided Asynchronous Split Federated Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i20.35503},
}
Figure