Abstract
Distributed machine learning (ML) training is a cornerstone of modern data center workloads, yet it often competes for resources with other applications. To optimize efficiency and utilization, training frameworks must be elastic—capable of scaling resources dynamically—and load-balanced. However, current elastic frameworks typically rely on "micro-tasks," which force high degrees of parallelism that can severely degrade algorithmic convergence. We introduce Chicle, an elastic distributed training framework that avoids the pitfalls of micro-tasks by utilizing a "uni-task" model combined with mobile data chunks. Our evaluation shows that Chicle maintains the algorithmic efficiency of rigid frameworks while providing the flexibility needed for shared, heterogeneous environments.
The Parallelism Trap in Distributed ML
In multi-tenant data centers, ML training jobs are frequently scaled up or down based on resource availability. Most existing elastic frameworks borrow techniques from general-purpose data processing (like Apache Spark), breaking work into thousands of small, independent "micro-tasks." While this approach provides scheduling flexibility, it creates a "parallelism trap" for ML algorithms like Stochastic Gradient Descent (SGD).
As parallelism increases—for instance, by using larger batch sizes or more data partitions—the model often requires more passes (epochs) over the data to reach the same level of accuracy. This algorithmic penalty means that while the system might process data faster in terms of throughput, the actual time to convergence may increase.
For example, doubling the number of partitions in the CoCoA algorithm on the Criteo dataset can increase the epochs needed for convergence by up to 65%. Micro-tasks force a high degree of parallelism to maintain scheduling granularity, inadvertently slowing down the learning process.
The Chicle Framework
To address these bottlenecks, we developed Chicle (named after the elastic latex in chewing gum). Chicle decouples scheduling granularity from the degree of algorithmic parallelism through three core innovations:
1. Uni-tasks
Instead of many micro-tasks, Chicle runs exactly one multi-threaded "uni-task" per node. This allows the framework to keep the total degree of data parallelism as low as possible for the current cluster size, maximizing the progress made per epoch.
2. Mobile Data Chunks
The unit of scheduling in Chicle is the "data chunk"—a small, fixed-size partition of the training set. By making the data mobile rather than the tasks, Chicle can redistribute work across the cluster without increasing the number of active tasks.
3. Iterative Scheduling and RDMA
ML training is inherently iterative, consisting of local model updates followed by global synchronization. Chicle exploits the natural pauses between these iterations to move data chunks.
To ensure that moving data doesn't become a bottleneck, Chicle utilizes Remote Direct Memory Access (RDMA). This allows for zero-copy data transfers that bypass CPU overhead, making it feasible to rebalance the cluster between iterations.
Evaluation and Results
We evaluated Chicle across several benchmarks, including HIGGS (SVM), CRITEO (linear models), and CIFAR-10/FASHION-MNIST (CNNs).
- Algorithmic Efficiency: In "scale-in" scenarios where nodes are removed, Chicle's uni-task approach converged significantly faster than micro-task emulations. Because it could dynamically reduce its batch size, it maintained higher progress per epoch.
- Dynamic Load Balancing: In heterogeneous clusters with mixed hardware, Chicle's scheduler "learns" node performance and proactively shifts data chunks to faster nodes. This reduced iteration times by approximately 20% compared to static, non-balanced systems.
- Performance Parity: Despite its flexibility, Chicle remains competitive with "rigid" state-of-the-art frameworks. It achieved convergence rates nearly identical to PyTorch (for CNNs) and Snap ML (for linear models), often with faster wall-clock times due to its efficient C++ implementation.
Conclusion
Chicle demonstrates that elasticity in ML does not have to come at the cost of algorithmic convergence. By moving data instead of tasks and respecting the iterative nature of ML, Chicle provides a path toward highly efficient, shared-resource training. Future work will explore automatic tuning of parallelism levels and mitigating synchronous bottlenecks caused by extreme stragglers.
Summary updated