Abstract
Cloud computing provides a powerful yet low-cost environment for distributed deep learning workloads. However, training complex deep learning models often requires accessing large amounts of data, which can easily exceed the capacity of local disks. Prior research often overlooks this training data problem by implicitly assuming that data is available locally or via low latency network-based data storage. Such implicit assumptions often do not hold in a cloud-based training environment, where deep learning practitioners create and tear down dedicated GPU clusters on demand, or do not have the luxury of local storage, such as in serverless workloads. In this work, we investigate the performance of distributed training that leverages training data residing entirely inside cloud storage buckets. These buckets promise low storage costs, but come with inherent bandwidth limitations that make them seem unsuitable for an efficient training solution. To account for these bandwidth limitations, we propose the use of two classical techniques, namely caching and pre-fetching, to mitigate the training performance degradation. We implement a prototype, DELI, based on the popular deep learning framework PyTorch by building on its data loading abstractions. We then evaluate the training performance of two deep learning workloads using Google Cloud's NVIDIA K80 GPU servers and show that we can reduce the time that the training loop is waiting for data by 85.6%-93.5% compared to loading directly from a storage bucket - thus achieving comparable performance to loading data directly from disk - while only storing a fraction of the data locally at a time. In addition, DELI has the potential of lowering the cost of running a training workload, especially on models with long per-epoch training times.
Introduction
In distributed deep learning, data access is frequently the primary bottleneck. Conventional training setups assume that massive datasets are either replicated locally on every GPU node or accessible via high-speed, low-latency network file systems (NFS). However, in modern cloud environments—particularly when using on-demand GPU clusters or serverless architectures—these assumptions are often invalid. Local storage is frequently too small or ephemeral, and configuring a dedicated high-performance NFS is both expensive and complex.
Cloud object storage (e.g., Google Cloud Storage buckets) offers a natural alternative: it is cost-effective, globally accessible, and infinitely scalable. However, these services suffer from high latency and strict bandwidth limitations. If a GPU must wait for a bucket to deliver the next batch of data, training time and associated costs increase significantly.
System Design: The DELI Framework
To address these challenges, we developed DELI, a prototype system built on top of PyTorch’s data loading abstractions. DELI bridges the gap between slow cloud buckets and fast GPUs using two primary strategies:
- Disk-Based Caching: Instead of relying on limited system RAM, DELI utilizes a local MongoDB instance on each node to cache data samples on the node's local disk. This ensures that once a sample is downloaded, it is available for subsequent epochs without further network overhead.
- Asynchronous Pre-fetching: DELI operates a separate service that "looks ahead" to identify future data requirements. While the GPU processes the current batch, the pre-fetcher concurrently downloads subsequent batches from the cloud bucket.
- The "50/50 Approach": We identified an optimal configuration where the fetch size and pre-fetch threshold are both set to 50% of the local cache size. This maintains a steady flow of data, minimizing cache misses by replenishing the cache just as it begins to deplete.
Experimental Evaluation
We evaluated DELI using two standard benchmarks representing different computational profiles:
- MNIST (Simple CNN): An I/O-bound workload where rapid model training puts maximum pressure on the data loader.
- CIFAR-10 (ResNet-50): A compute-bound workload where longer batch processing times provide more opportunity for background pre-fetching.
Key Results
DELI demonstrates substantial performance gains over direct cloud bucket access. For the MNIST workload, it reduced data loading wait times by 85.6%, and for CIFAR-10, the reduction reached 93.5%.
Crucially, DELI achieved data loading speeds nearly identical to having the entire dataset stored on a local disk, despite only keeping a small fraction of the data locally at any given time.
Discussion and Insights
Our findings reveal several key insights into distributed training with cloud storage:
- Complexity Advantage: DELI performs exceptionally well on complex models like ResNet-50. The increased computation time per batch provides the pre-fetching service with more "breathing room" to hide cloud bucket latency.
- The Necessity of Pre-fetching: In distributed environments where nodes receive random data subsets each epoch, caching alone is insufficient. Pre-fetching is the critical component that ensures data availability.
- Cost Efficiency: While cloud buckets incur API request fees, the reduction in GPU rental time often results in lower overall costs. For the ResNet/CIFAR-10 workload, the "50/50 approach" proved more cost-effective than baseline disk-based training by optimizing the balance between storage and compute expenses.
Limitations and Future Work
While promising, DELI's current implementation has limitations. Future work will explore scaling to multi-terabyte datasets and optimizing API costs through "super-sampling" (grouping multiple data points into single files). Additionally, while tested on Google Cloud, the underlying logic is designed to be architecture-agnostic and applicable to other providers like AWS or Azure.
Summary updated