WRATH: Workload Resilience Across Task Hierarchies in Task-based Parallel Programming Frameworks

Sicheng Zhou, Zhuozhao Li, Valérie Hayot-Sasson, Haochen Pan, Maxime Gonthier, J. Gregory Pauloski, Ryan Chard, Kyle Chard, Ian T. FosterOpenReview 2025 · DOI · arXiv · OpenReview

Abstract

Failures in Task-based Parallel Programming (TBPP) can severely degrade performance and result in incomplete or incorrect outcomes. Existing failure-handling approaches, including reactive, proactive, and resilient methods such as retry and checkpointing mechanisms, often apply uniform retry mechanisms regardless of the root cause of failures, failing to account for the unique characteristics of TBPP frameworks such as heterogeneous resource availability and task-level failures. To address these limitations, we propose WRATH, a novel systematic approach that categorizes failures based on the unique layered structure of TBPP frameworks and defines specific responses to address failures at different layers. WRATH combines a distributed monitoring system and a resilient module to collaboratively address different types of failures in real time. The monitoring system captures execution and resource information, reports failures, and profiles tasks across different layers of TBPP frameworks. The resilient module then categorizes failures and responds with appropriate actions, such as hierarchically retrying failed tasks on suitable resources. Evaluations demonstrate that WRATH significantly improves TBPP robustness, tripling the task success rate and maintaining an application success rate of over 90% for resolvable failures. Additionally, WRATH can reduce the time to failure by 20%-50%, allowing tasks that are destined to fail to be identified and fail more quickly.

Introduction

Task-based Parallel Programming (TBPP) frameworks, such as Dask, Ray, and Parsl, have become indispensable for scaling complex computational workloads across distributed systems. However, as these systems scale, the frequency of failures—ranging from transient network glitches to permanent hardware faults—increases significantly. Traditional resilience strategies often rely on "blind" retries or periodic checkpointing. These methods are frequently inefficient; for instance, retrying a task that failed due to a fundamental code error wastes valuable computational resources, while retrying a memory-intensive task on the same resource-constrained node is destined for repeated failure.

The WRATH (Workload Resilience Across Task Hierarchies) framework addresses these inefficiencies by introducing context-aware resilience. By understanding the root cause of a failure, WRATH can apply targeted recovery policies, such as "failing fast" for unrecoverable errors or reallocating tasks to nodes with appropriate resources.

The WRATH Framework

The core of WRATH is a systematic approach to failure management built upon three pillars: layered categorization, distributed monitoring, and an intelligent resilience module.

Layered Failure Categorization

WRATH organizes the TBPP stack into four distinct layers to better identify failure origins:

  • Application Layer: Captures user-level bugs, such as syntax errors or logic flaws.
  • Framework Layer: Monitors issues within the orchestrator, such as scheduler failures.
  • Runtime Layer: Identifies resource-related issues, including memory exhaustion or CPU starvation.
  • Environment Layer: Tracks infrastructure failures, such as hardware shutdowns or library mismatches.

Intelligent Resilience Module

The Resilience Module utilizes a Failure Taxonomy Library (FTL) to map error messages and system logs to specific root causes. Based on this categorization, it executes a tailored policy.

The WRATH Resilience Policy Engine categorizes failures to determine the optimal recovery action, such as hierarchical retries or immediate termination.

As shown in the flowchart above, the engine distinguishes between non-resolvable failures (leading to immediate termination) and retriable failures. For resource-related issues, WRATH employs Hierarchical Retry, which reallocates the task to a different resource pool rather than simply repeating the execution on the same node.

Experimental Evaluation

The effectiveness of WRATH was evaluated using the TaPS benchmark suite, comprising five real-world scientific applications: Cholesky (linear algebra), Docking (molecular modeling), FedLearn (federated learning), MapReduce, and MolDesign (molecular design). A custom failure-injection engine was used to simulate a variety of errors, including ulimit violations, memory exhaustion, and environment mismatches.

Key Results

The implementation of WRATH yielded significant improvements in both robustness and efficiency:

  • Improved Success Rates: WRATH demonstrated a dramatic increase in task success rates, particularly in heterogeneous environments. In comparative tests against a standard Parsl baseline, WRATH maintained high success rates even as the number of nodes increased.
Comparison of success rates between Parsl with and without the WRATH framework across varying node counts.
  • Efficiency through "Fail Fast": By identifying application-layer bugs immediately, WRATH avoids useless retries, reducing the "time to failure" by 20% to 50%. This allows developers to receive feedback faster and saves significant compute time.
  • Low Operational Overhead: Despite the continuous monitoring and real-time log analysis, WRATH introduces less than 2% overhead to total execution time, making it suitable for production-scale deployments.

Conclusion

WRATH represents a significant step forward in making TBPP frameworks resilient to the complexities of large-scale distributed computing. By moving from uniform retries to context-aware recovery, it maximizes resource utilization and improves application reliability. Future research will focus on extending WRATH to support compiled languages like C++ and Fortran, as well as enhancing cross-language interoperability for multi-language workflows.

Summary updated

Abstract

Failures in Task-based Parallel Programming (TBPP) can severely degrade performance and result in incomplete or incorrect outcomes. Existing failure-handling approaches, including reactive, proactive, and resilient methods such as retry and checkpointing mechanisms, often apply uniform retry mechanisms regardless of the root cause of failures, failing to account for the unique characteristics of TBPP frameworks such as heterogeneous resource availability and task-level failures. To address these limitations, we propose WRATH, a novel systematic approach that categorizes failures based on the unique layered structure of TBPP frameworks and defines specific responses to address failures at different layers. WRATH combines a distributed monitoring system and a resilient module to collaboratively address different types of failures in real time. The monitoring system captures execution and resource information, reports failures, and profiles tasks across different layers of TBPP frameworks. The resilient module then categorizes failures and responds with appropriate actions, such as hierarchically retrying failed tasks on suitable resources. Evaluations demonstrate that WRATH significantly improves TBPP robustness, tripling the task success rate and maintaining an application success rate of over 90% for resolvable failures. Additionally, WRATH can reduce the time to failure by 20%-50%, allowing tasks that are destined to fail to be identified and fail more quickly.
BibTeX
@misc{zhou2025wrath,
  author = {Sicheng Zhou and Zhuozhao Li and Valérie Hayot-Sasson and Haochen Pan and Maxime Gonthier and J. Gregory Pauloski and Ryan Chard and Kyle Chard and Ian T. Foster},
  title = {{WRATH: Workload Resilience Across Task Hierarchies in Task-based Parallel Programming Frameworks}},
  year = {2025},
  eprint = {2503.12752},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.12752},
  howpublished = {OpenReview},
}
Figure