TL;DR
This paper introduces a novel approach using Mixture Density Networks (MDNs) to estimate the resource usage (CPU and memory) of data stream processing workloads. Unlike previous methods that provided single-point estimates, this technique predicts a full probability distribution, offering a more comprehensive understanding of resource demands and enabling better workload management in cloud environments.
What Problem Does This Paper Address?
Cloud computing offers elastic resources for data stream processing, but effectively managing these resources requires accurate predictions of workload demands. Traditional methods for estimating resource usage, particularly for SQL or OLTP workloads, often provide a single value (e.g., average CPU usage). However, data stream processing is characterized by uncertain arrival patterns, diverse query types, and fluctuating resource consumption. This "multi-modal nature" means a single point estimate is insufficient and can lead to inefficient resource allocation, over-provisioning, or performance degradation. The challenge is to accurately predict resource needs that can vary significantly, even for the same input.
Methods
The core of the proposed solution is a Mixture Density Network (MDN). An MDN is a hybrid model that combines a neural network with a mixture model (specifically, a Gaussian Mixture Model or GMM in this work).
Here's how it works:
- Input Features: The system takes features extracted from continuous query language (CQL) statements and query plans as input. These include aspects like arrival rates, number of data streams, subqueries, aggregation functions, join predicates, project size, and window types/sizes.
- Neural Network: A feed-forward neural network processes these input features.
- Mixture Model Parameters: Instead of directly outputting a resource usage value, the neural network outputs parameters for a GMM. These parameters define a probability distribution, specifically the mixture weights (how much each Gaussian component contributes), the means (centers of the Gaussians), and the variances (spread of the Gaussians).
- Probability Distribution Output: The GMM then generates a probability density function (PDF) for the target resource usage (e.g., CPU or memory). This PDF represents the entire spectrum of possible resource demands, not just a single average.
This approach is novel because it moves beyond single-point estimations to provide a probabilistic forecast, which is crucial for capturing the inherent variability and multi-modal nature of data stream processing workloads.
Datasets
The authors evaluated their approach using two primary datasets/benchmarks:
- Linear Road Benchmark (LRB): This is a standard benchmark for comparing the performance of streaming systems. The authors treated its 20 queries as templates and generated 17,289 execution traces by varying arrival rates and window sizes.
- TPC-H Benchmark: While originally designed for databases, the authors adapted TPC-H for stream processing. They generated 35 executable queries and 8,783 performance traces. This adaptation involved treating relations as data streams and modifying queries to include time windows, making it more suitable for stream processing evaluation.
Key Insights and Ideas
- Probabilistic Prediction is Crucial: The paper strongly argues that for data stream processing, single-point resource estimates are inadequate. The inherent variability and multi-modal nature of these workloads necessitate a probabilistic approach to capture the full range of resource demands.
- MDNs Excel at Multi-Modal Data: MDNs are well-suited for modeling complex, multi-modal distributions, which are common in stream processing due to factors like windowing mechanisms.
- Feature Engineering Matters: The accuracy of the MDN model is highly dependent on the quality and relevance of the input features extracted from query statements and plans.
- CPU vs. Memory Prediction: The study observed that memory prediction models converged faster and achieved lower error metrics than CPU prediction models, potentially due to a simpler input feature set for memory.
- Benchmark Adaptation is Key: Adapting existing benchmarks like TPC-H for stream processing is essential for robust evaluation, but requires careful consideration of stream-specific aspects.
Key Results
The MDN approach demonstrated high accuracy across various error metrics:
- Continuous Ranked Probability Score (CRPS): This metric evaluates the overall accuracy of the predicted PDF. The paper reports small CRPS values for both LRB and TPC-H workloads, indicating good prediction accuracy.
- Negative Log Predictive Density (NLPD): This metric assesses how well the model assigns probability to the actual observations. Low NLPD scores were achieved, suggesting the model is "honest" in its probabilistic assessments.
- Mean-Square Error (MSE): While the primary focus is on probabilistic prediction, MSE was also used. The results showed small MSE values, indicating that the predicted distributions' central tendencies (medians) were close to actual values.
For instance, in the LRB workload, the MDN achieved CRPS values around 0.113-0.128 for CPU and 0.065-0.066 for memory, with corresponding NLPD scores around -0.865 and -1.465, respectively. The TPC-H results were also competitive, with CRPS around 0.154-0.171 for CPU and 0.055-0.097 for memory.
Limitations and Future Work
- No Concurrent Execution: The current model does not consider resource consumption in the presence of concurrent executions or combined workloads. This is a significant limitation, as real-world systems often handle multiple queries simultaneously.
- Centralized System Focus: The study focused on centralized stream processing systems. Extending the approach to distributed systems is a natural next step.
- Standardization of Query Languages: The lack of a universal standard for continuous query languages presents challenges, requiring the model to be generalized across common operator sets.
- Future Work: The authors suggest investigating how to deploy this prediction approach within workload management strategies, exploring its impact on SLA specifications, and extending it to handle concurrent executions and resource sharing in distributed stream processing environments.
Summary updated