TL;DR
This paper introduces Litz, an elastic framework designed to make distributed machine learning (ML) applications adaptable to the dynamic resource availability of cloud environments. Litz achieves this by providing a novel programming model that supports stateful workers, model scheduling, and relaxed consistency, enabling ML jobs to scale in and out seamlessly without sacrificing performance.
What Problem Does This Paper Address?
Modern cloud and data-center environments are characterized by constantly changing resource availability due to their multi-tenant nature. Machine learning (ML) applications are increasingly prevalent in these environments, but existing distributed ML frameworks are often "inelastic." This means they cannot dynamically adapt to fluctuating resources, leading to inefficient resource utilization and potential performance bottlenecks. Traditional general-purpose distributed frameworks like Hadoop and Spark are not well-suited for the unique demands of ML, resulting in significant performance degradation. Consequently, ML applications are often deployed with rigid resource allocations, unable to take advantage of available idle resources or gracefully release them when needed. This paper addresses the critical need for an elastic ML framework that can support advanced ML techniques while seamlessly adapting to dynamic resource landscapes.
Methods
Litz introduces a novel programming model and an optimized execution system to achieve elasticity for distributed ML.
- Event-Driven Programming Model: Litz separates applications from the physical cluster through an event-driven API. This allows for stateful workers and transparent management of application state and computation during elastic events (scaling up or down). Computation is broken down into "micro-tasks" that access a distributed parameter server.
- Task-Driven Consistency Model: Litz supports model scheduling by allowing micro-tasks to be defined with dependencies. This enables applications to specify the order of model updates. It also implements a "causal memory" consistency model, where micro-tasks observe updates from their dependencies but have intentionally weak guarantees between independent tasks, facilitating relaxed consistency.
- Optimized Elastic Execution System: Litz's execution system transparently re-balances workload during scaling events without requiring explicit application intervention. It leverages Litz's programming and consistency models to minimize system overhead, aiming for performance comparable to non-elastic systems. Key optimizations include:
- Parameter Cache Synchronization: Caching parameter values locally to reduce network calls, especially effective with relaxed consistency.
- Parameter Update Aggregation: Aggregating local updates before sending them to the parameter server to reduce network traffic.
- Co-operative Multitasking: Using co-routines to allow worker threads to switch to other tasks when one is blocked, overlapping communication and computation.
- Consistent Checkpoint and Recovery: Periodically saving application state to enable automatic recovery from failures.
Datasets
- ImageNet ILSVRC2012: Used for training and evaluating the Multinomial Logistic Regression (MLR) application. The dataset was pre-processed to 81GB with 21K features per image.
- ClueWeb12: Used for training and evaluating the Latent Dirichlet Allocation (LDA) application. A subsample of 50 million English web pages was used, resulting in 10 billion tokens and 2 million distinct words, totaling 88GB.
- CIFAR-10: Used for evaluating deep neural networks (DNNs).
The paper does not introduce new datasets but uses standard, large-scale datasets to evaluate its framework.
Key Insights and Ideas
- Elasticity is Crucial for Modern ML: The paper strongly argues that ML applications, due to their iterative nature and potential for resource variability, are prime candidates for elastic execution in dynamic cloud environments.
- Bridging the Gap Between ML Techniques and Elasticity: A core insight is that existing advanced ML techniques (stateful workers, model scheduling, relaxed consistency) are not inherently incompatible with elasticity. Litz's programming model is designed to expose these techniques in a way that the execution system can leverage for elastic adaptation.
- Performance without Compromise: A significant finding is that Litz can achieve elasticity with minimal performance overhead compared to state-of-the-art non-elastic ML frameworks. This is achieved through clever optimizations like parameter caching and update aggregation.
- Resource Variability as an Opportunity: The paper highlights that ML algorithms often exhibit dynamic memory usage patterns. Litz can expose these patterns, allowing for more efficient resource allocation and potentially enabling novel scheduling policies.
- Micro-tasking for Elasticity: Decomposing computations into short-lived micro-tasks is key to Litz's ability to gracefully migrate state and computation during scaling events, as it provides frequent opportunities for the system to intervene.
Key Results
- MLR Performance: Litz's MLR implementation achieved convergence about 8x faster than Bösen, a non-elastic ML system.
- LDA Performance: Litz's LDA implementation was competitive with Strads, a specialized non-elastic ML system, completing epochs about 6% slower but achieving a better objective value.
- Elastic Scale-Out:
- MLR scaling out from 4 to 8 nodes resulted in a 22% shorter total run-time.
- LDA scaling out from 12 to 24 nodes resulted in a 27% shorter total run-time.
- Litz's scale-out performance was very close to the "ideal" scale-out time, with less than 1% difference for MLR and 5% for LDA.
- Elastic Scale-In:
- MLR scale-in took an average of 2.5 seconds.
- LDA scale-in took an average of 43 seconds.
- These scale-in times are significantly faster than the warning times provided by preemptive cloud environments.
- Priority Scheduling: In a priority scheduling scenario, Litz enabled an LDA job to scale in, allowing a higher-priority job to run without killing and restarting the LDA job, leading to a much shorter overall completion time for the LDA job.
- Deep Neural Networks: Litz trained a deep feed-forward network 3.4x faster than TensorFlow.
Limitations and Future Work
- Limitations:
- The paper acknowledges that LDA's higher overhead during scale-out is due to its inherent large worker state, which needs serialization. This suggests that stateful applications might still incur some transfer costs.
- While Litz's performance is competitive, there's still room for optimization, particularly in reducing state transfer overhead for stateful applications.
- Future Work:
- Resource Usage Patterns: Further investigation into the dynamic memory usage patterns of ML applications to develop more sophisticated scheduling policies that leverage this variability.
- Deep Learning and Elastic GPU Computing: Exploring how Litz's event-driven API can integrate with higher-level symbolic programming models of deep learning frameworks (e.g., TensorFlow, MXNet) for elastic GPU execution.
- Compiler Integration: Investigating the role of frameworks like Litz in the elastic and efficient execution of future deep learning applications, especially with the trend towards compiler techniques for separating programming and execution.
Summary updated