Incorporating Serverless Computing into P2P Networks for ML Training: In-Database Tasks and Their Scalability Implications (Student Abstract)

Amine BarrakAAAI 2024 · DOI

Abstract

Distributed Machine Learning (ML) often faces a trade-off between the efficiency of centralized parameter servers and the robustness of Peer-to-Peer (P2P) networks. This paper introduces a hybrid architecture designed to enhance decentralized ML by integrating serverless computing and in-database processing. By offloading gradient computations to stateless serverless functions and performing model aggregations and updates directly within a Redis database (RedisAI), the proposed framework mitigates common resource constraints and communication bottlenecks. Experimental results demonstrate up to an 82% reduction in model update times and substantial improvements in training throughput compared to traditional instance-based P2P approaches.

Introduction

Training large-scale ML models across P2P networks is highly desirable due to the lack of a single point of failure and inherent scalability. However, practical implementation is often hindered by two primary challenges:

  1. Resource Constraints: Edge devices or local workstations within a P2P network frequently lack the memory and compute capacity required for efficient parallel batch processing.
  2. Communication Overhead: While serverless computing offers a scalable compute solution, its stateless nature typically necessitates moving large gradient tensors between functions and storage, creating a significant network bottleneck.

Proposed Architecture

The proposed framework addresses these issues by linking each peer to a dedicated Redis database and a suite of serverless functions. The architecture focuses on two dimensions of scalability: intra-peer scalability (parallelizing local workloads) and inter-peer scalability (managing global synchronization).

Figure 2: Overview of the hybrid P2P architecture featuring serverless gradient computation and in-database aggregation.

As illustrated in Figure 2, the training process involves:

  • Serverless Offloading: Datasets are partitioned into batches and processed by multiple serverless functions to compute gradients in parallel.
  • In-Database ML: To eliminate the overhead of transferring large tensors, the "Average Gradient" and "Update Model" tasks are executed inside the Redis database using RedisAI. This "compute-to-data" approach ensures that massive gradient tensors do not need to leave the database environment for aggregation.
  • Orchestration: A state machine service manages the training epoch transitions and monitors model convergence.

Experimental Evaluation

The framework was evaluated using the MNIST dataset across three model architectures: VGG11, DenseNet-121, and ResNet-18. Performance was compared between traditional instance-based training (Without Serverless - WoS) and the proposed serverless-enhanced approach (With Serverless - WS), as well as between in-database and external processing.

Key Results and Discussion

The quantitative improvements are significant across several metrics. For a VGG11 model with a batch size of 64, the serverless approach (WS) achieved a training time of 10.5 seconds, compared to 394.8 seconds for the traditional approach—a 38x speedup.

The efficiency of in-database processing is particularly noteworthy. By performing updates within the database, the system avoids the serialization and network transfer of large tensors.

Figure 5: Comparison of average and update times for in-database (W. DB) versus outside-database (O. DB) processing.

As shown in Figure 5, performing model updates inside the database (W. DB) consistently took only 4.8 seconds, regardless of batch size, representing an 82% reduction compared to the 27.5 seconds required for external processing (O. DB). Similarly, gradient averaging for large batch sizes (1024) saw a 44% reduction in time when performed in-database.

However, these performance gains come with a "pay-for-speed" trade-off. The serverless approach is more expensive than running dedicated instances. Additionally, while inter-peer scaling is beneficial, it is eventually limited by the network communication required for global synchronization.

Conclusion

This research demonstrates that integrating serverless computing and in-database processing can dramatically accelerate decentralized ML training. By moving computation to the data, the framework overcomes the traditional bottlenecks of stateless serverless functions. Future work will focus on optimizing function invocation costs and improving synchronization protocols to further enhance inter-peer scalability.

Summary updated

BibTeX
@inproceedings{barrak2024incorporating,
  author = {Amine Barrak},
  title = {{Incorporating Serverless Computing into P2P Networks for ML Training: In-Database Tasks and Their Scalability Implications (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30419},
}
Figure