TL;DR
This paper tackles the challenge of efficiently deploying data-intensive applications across multiple cloud providers. They introduce a novel two-stage approach that first intelligently selects the best cloud providers based on user needs and costs, and then optimizes task placement to minimize data movement and speed up execution. This is crucial for making large-scale cloud computing more cost-effective and performant.
What Problem Does This Paper Address?
Migrating large-scale, data-intensive applications to the cloud offers significant benefits, but choosing the right cloud provider(s) and managing resources effectively is a complex, often manual, and NP-hard problem. Users face a bewildering array of services, pricing models, and Service Level Agreements (SLAs) across different cloud providers. Furthermore, for data-intensive applications, the cost and time associated with transferring data between clouds (intercloud data transfers) can be substantial, impacting both performance and budget. Existing solutions often focus on optimizing only a few objectives (like cost or makespan) and struggle to handle the multi-dimensional nature of user requirements and the complexities of multi-cloud environments. This paper aims to automate this process by providing a holistic resource allocation strategy.
Methods
The paper proposes a two-stage resource allocation scheme:
- SLA-based Matching: This stage selects the most suitable cloud providers.
- Sieving Matching: A basic algorithm that randomly selects datacenters satisfying functional and non-functional SLA requirements and budget constraints.
- Utility-based Matching: A more sophisticated approach inspired by economic theory. It uses a quasi-linear utility function to maximize user profit by considering user preferences for non-functional SLA attributes (like availability, latency, throughput) and their willingness to pay. This algorithm uses a genetic algorithm to search for the optimal cloud composition.
- Data Locality Driven Scheduling: Once clouds are selected, this stage distributes application workloads.
- DAS (Data-Aware Size-based) Scheduler: Assigns tasks to VMs based on the size of required input files already present in a datacenter, prioritizing datacenters with the largest amount of local data.
- DAT (Data-Aware Time-based) Scheduler: Assigns tasks by calculating the time needed to transfer missing input files to each datacenter, prioritizing datacenters that minimize transfer time.
The framework uses a broker-based workflow engine and integrates with CloudSim for simulation.
Datasets
The paper evaluates their approach using a real data-intensive workflow application: Montage. This workflow is used to construct large image mosaics of the sky. The authors imported a real XML-formatted Montage trace executed on the FutureGrid testbed using the Pegasus WfMS. This trace contains 7463 tasks across 11 horizontal levels, with 3 GB of input data and generating significant intermediate and traffic data. The evaluation uses synthetic Cloud-to-Cloud latency and bandwidth values due to the lack of readily available real-world metrics.
Key Insights and Ideas
- Utility Maximization is Key: The utility-based matching algorithm significantly outperforms simpler methods by truly capturing user preferences and willingness to pay, leading to better overall outcomes.
- Data Locality is Crucial for Performance: Scheduling strategies that actively consider data location (like DAS and DAT) dramatically reduce intercloud data transfers, which directly impacts execution time and cost, especially in scenarios where clouds are geographically dispersed.
- Two-Stage Approach Pays Off: Separating the cloud selection (matching) from task placement (scheduling) allows for specialized optimization at each stage, leading to a more effective overall allocation.
- Clustering Trade-offs: While clustering workflows can reduce scheduling overhead, increasing the number of clusters can negatively impact makespan. The paper found a sweet spot for optimal performance.
Key Results
- The proposed multi-dimensional resource allocation scheme (utility-based matching + DAS/DAT scheduler) achieved the lowest execution time (makespan) compared to other combinations and baseline schedulers (Round Robin, Min-Min).
- The DAS scheduler was particularly effective in keeping Intercloud transfer size under 10 GB for both scenarios tested.
- The data locality driven schedulers (DAS and DAT) reduced the transfer time ratio significantly, up to 37% and 50% for the EU and EU-US scenarios, respectively.
- The utility-based matching combined with DAS scheduler showed significant cost savings, up to 25% for compute and 15% for traffic costs compared to Sieving.
- The paper demonstrates that the utility algorithm tends to deploy VMs on the cheapest datacenters, leading to cost savings, especially when clouds are geographically close.
Limitations and Future Work
- Simulation-based Evaluation: The evaluation is primarily based on a simulation environment (CloudSim), not real-world cloud deployments. While this allows for controlled experiments, real-world performance might differ.
- Synthetic Network Metrics: The study uses defined synthetic values for Cloud-to-Cloud latency and bandwidth, as real-world metrics are not easily accessible.
- Limited Workflow Types: The evaluation focused on the Montage workflow. Future work will explore other types of large-scale applications, such as MapReduce.
- Automated SLA Metric Collection: The authors plan to automate the collection of real-time SLA metrics from public clouds.
- Adaptive Policies: Investigating adaptive matching and scheduling policies to handle resource and network failures in cloud environments is a future direction.
Summary updated