Optimization towards Efficiency and Stateful of dispel4py

Liang Liang, H. Zhang, Guang Yang, Thomas Heinis, Rosa Filgueira

Abstract

Scientific workflows bridge scientific challenges with computational resources. While dispel4py, a stream-based workflow system, offers mappings to parallel enactment engines like MPI or Multiprocessing, its optimization primarily focuses on dynamic process-to-task allocation for improved performance. An efficiency gap persists, particularly with the growing emphasis on conserving computing resources. Moreover, the existing dynamic optimization lacks support for stateful applications and grouping operations. To address these issues, our work introduces a novel hybrid approach for handling stateful operations and groupings within workflows, leveraging a new Redis mapping. We also propose an auto-scaling mechanism integrated into dispel4py's dynamic optimization. Our experiments showcase the effectiveness of auto-scaling optimization, achieving efficiency while upholding performance. In the best case, auto-scaling reduces dispel4py's runtime to 87% compared to the baseline, using only 76% of process resources. Importantly, our optimized stateful dispel4py demonstrates a remarkable speedup, utilizing just 32% of the runtime compared to the contender.

Introduction

Scientific workflows are essential for processing massive data streams in distributed environments. dispel4py is a Python-based framework designed for such data-intensive streaming workflows, providing the ability to map abstract workflows to various parallel engines. However, two significant limitations have hindered its broader adoption:

  1. Statefulness: Real-world applications often require Processing Elements (PEs) to maintain state across inputs (e.g., running averages). Existing dynamic optimizations in dispel4py struggle to maintain this state when tasks are randomly assigned to processes.
  2. Resource Efficiency: Standard deployments often use static resource allocation, leading to "busy-wait" states where CPU cycles are consumed even when no data is being processed, resulting in unnecessary energy consumption and costs.

Methodology

To address these challenges, we introduce a hybrid mapping strategy and an auto-scaling engine within the dispel4py architecture.

Overview of the dispel4py architecture, highlighting the mapping and optimization modules.

Hybrid Redis Mapping

We replaced standard multiprocessing queues with Redis Streams. To support stateful operations, we developed a "hybrid" strategy. Stateless tasks are pulled from a global queue for maximum flexibility. In contrast, stateful tasks are "pinned" to dedicated processes with Private Queues. This ensures that data requiring specific state information always reaches the correct process in the correct sequence without the overhead of global synchronization.

Auto-scaling Engine

An Auto_scaler class was integrated to monitor system load in real-time. For Multiprocessing environments, it monitors queue size to wake up idle processes as demand increases. For Redis-based environments, it monitors average idle time, transitioning processes into a low-energy standby mode when they are underutilized. Additionally, a "poison pill" retry mechanism was introduced to ensure system termination only occurs when the workflow is truly complete, rather than just experiencing a temporary slowdown.

Evaluation Workflows

The system was evaluated using three distinct scientific workflows:

  • Internal Extinction of Galaxies: An astrophysics workflow for measuring optical luminosity, used to evaluate stateless performance.
  • Seismic Cross-Correlation: A geological workflow for earthquake monitoring, testing grouping operations.
  • Sentiment Analysis for News Articles: A workflow tracking sentiment over time using news datasets, serving as the primary test for stateful PEs.

Results and Discussion

Our evaluation demonstrates significant improvements in both efficiency and performance:

  • Efficiency: For stateless workflows, the auto-scaling mechanism achieved 87% of the baseline runtime while consuming only 76% of the process resources.
  • Stateful Performance: The hybrid_redis mapping outperformed the standard multi baseline significantly, completing stateful applications in just 32% of the original runtime.
  • Portability: The system's effectiveness was validated across local servers, Google Cloud, and High-Performance Computing (HPC) environments.

The results highlight a crucial distinction between "Runtime" and "Process Time." While auto-scaling may introduce minor overhead when waking processes, the reduction in total CPU hours (Process Time) leads to substantial savings in cloud costs and energy.

Conclusion

This work enhances dispel4py by introducing stateful support and resource efficiency through hybrid Redis mapping and auto-scaling. While current scaling logic is incremental, future work will explore predictive auto-scaling using Machine Learning to further minimize performance penalties during sudden workload spikes.

Summary updated

Abstract

Scientific workflows bridge scientific challenges with computational resources. While dispel4py, a stream-based workflow system, offers mappings to parallel enactment engines like MPI or Multiprocessing, its optimization primarily focuses on dynamic process-to-task allocation for improved performance. An efficiency gap persists, particularly with the growing emphasis on conserving computing resources. Moreover, the existing dynamic optimization lacks support for stateful applications and grouping operations. To address these issues, our work introduces a novel hybrid approach for handling stateful operations and groupings within workflows, leveraging a new Redis mapping. We also propose an auto-scaling mechanism integrated into dispel4py's dynamic optimization. Our experiments showcase the effectiveness of auto-scaling optimization, achieving efficiency while upholding performance. In the best case, auto-scaling reduces dispel4py's runtime to 87% compared to the baseline, using only 76% of process resources. Importantly, our optimized stateful dispel4py demonstrates a remarkable speedup, utilizing just 32% of the runtime compared to the contender.
BibTeX
@misc{liang2023optimization,
  author = {Liang Liang and H. Zhang and Guang Yang and Thomas Heinis and Rosa Filgueira},
  title = {{Optimization towards Efficiency and Stateful of dispel4py}},
  year = {2023},
  eprint = {2309.00595},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.00595},
}
Figure