Corgi$^2$: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGD
Lists
Only you see this
Abstract
When using Stochastic Gradient Descent (SGD) for training machine learning models, it is often crucial to provide the model with examples sampled at random from the dataset. However, for large datasets stored in the cloud, random access to individual examples can be costly and inefficient. A recent work proposed an online shuffling algorithm called CorgiPile, which greatly improves efficiency of data access, at the cost of some performance loss, which becomes particularly apparent for large datasets stored in homogeneous shards (e.g., video datasets). In this paper, we extend CorgiPile by adding an efficient offline iteration, transforming it into a hybrid two-step partial data shuffling strategy. We show through comprehensive theoretical analysis that our approach performs similarly to SGD with random access (even for homogenous data) without compromising on the data access efficiency of CorgiPile, and demonstrate its practical advantages through experimental results.
BibTeX
@misc{livne2023corgi,
author = {Etay Livne and Gal Kaplun and eran malach and Shai Shalev-Shwartz},
title = {{Corgi$^2$: A Hybrid Offline-Online Approach To Storage-Aware Data Shuffling For SGD}},
year = {2023},
eprint = {2309.01640},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2309.01640},
}