Scientific Workflow Systems for 21st Century, New Bottle or New Wine?

Yong Zhao, Ioan Raicu, Ian T. FosterSERVICES I 2008 · DOI · OpenReview

TL;DR

This paper explores the evolving landscape of scientific workflow systems (SWFS) in the face of massive data growth and the rise of multi-core processors. It argues that while existing workflow technologies are valuable, the unique demands of 21st-century e-science necessitate a re-evaluation and potentially new approaches to SWFS design, focusing on scalability, performance, and seamless integration with high-performance computing (HPC) and grid environments.

What Problem Does This Paper Address?

The paper addresses the fundamental question of whether current scientific workflow systems are adequate for the challenges of 21st-century e-science, or if entirely new paradigms are required. The authors highlight two major drivers of this challenge:

  1. The Data Deluge: Scientific datasets are growing exponentially, far outpacing computational resources. This necessitates efficient processing and analysis of massive amounts of data, often distributed across vast computational infrastructures.
  2. The Rise of Multi-core Processors: The stagnation of single-core processor clock speeds has led to the widespread adoption of multi-core architectures. This shift demands new approaches to software design and development, particularly in handling concurrency and parallelism, which are crucial for scientific computations.

The paper questions whether existing workflow technologies, developed in a different era, can effectively manage these new realities, or if they are akin to putting "new wine" (21st-century scientific challenges) into "old bottles" (existing SWFS).

Methods

The authors employ an analytical and survey-based approach. They:

  • Analyze the status and challenges of scientific workflows: This involves identifying the core aspects SWFS are engaged with, such as describing complex procedures, automating data derivation, enabling high-performance computing, and managing provenance.
  • Investigate existing technologies and emerging languages, platforms, and systems: They review established workflow coordination languages, business process management systems, and specific scientific workflow systems like DAGMan, Pegasus, Taverna, Kepler, and Triana. They also discuss emerging concepts like VisTrails for dataflow evolution and MapReduce for large-scale data processing.
  • Identify key challenges: Based on their analysis, they pinpoint the critical issues that future SWFS must address, including scalability, reliability, scheduling, monitoring, data management, collaboration, provenance, and workflow evolution.
  • Discuss opportunities: They explore how to leverage existing and emerging technologies, including multi-core architectures and grid computing, to meet these challenges.

Datasets

Not discussed in the paper. The paper focuses on the conceptual challenges and existing technologies rather than presenting empirical results based on specific datasets.

Key Insights and Ideas

  • The "new wine, old bottle" dilemma: The core idea is that the exponential growth of data and the shift to multi-core computing represent a fundamental change that may outstrip the capabilities of existing workflow systems designed for a different era.
  • Convergence of Supercomputing and Grid Computing: The increasing power and accessibility of multi-core architectures in supercomputers are blurring the lines with traditional grid computing, suggesting a need for SWFS that can leverage both.
  • Data locality is paramount: For data-intensive scientific applications, efficient data movement and caching are critical bottlenecks. SWFS need to be "data-aware" to minimize data transfer.
  • Domain-specific vs. Generic Platforms: There's a tension between the need for specialized languages that improve productivity for specific scientific domains and the demand for generic, scalable platforms that can manage large-scale scientific applications across diverse disciplines.
  • Implicit parallelism is desirable: For end-users, implicit parallelism (where the system discovers and schedules independent tasks) is preferred over explicit parallelism, which requires significant expertise. However, some level of explicit control may still be necessary.
  • Lack of standardized infrastructure: Unlike business process management, the scientific domain lacks agreed-upon standards and steering committees for workflow management, leading to reinvention of the wheel.

Key Results

This paper is primarily a position and survey paper, not one presenting novel experimental results. Therefore, there are no quantitative "key results" in terms of performance metrics or accuracy improvements. The "results" are conceptual:

  • Identification of key challenges: The paper clearly articulates the critical challenges for 21st-century scientific workflows, including data deluge, multi-core parallelism, scalability, and data management.
  • Overview of existing and emerging technologies: It provides a valuable survey of current SWFS, programming languages, and computing paradigms relevant to scientific computation.
  • A call for new approaches: The paper strongly advocates for the development of new SWFS or significant evolution of existing ones to meet the demands of modern e-science.

Limitations and Future Work

Limitations:

  • Lack of empirical evaluation: The paper is a conceptual analysis and survey, not an empirical study. It doesn't present new systems or benchmark existing ones against the identified challenges.
  • Focus on challenges, not solutions: While it identifies problems, it doesn't propose concrete solutions or new architectures for SWFS.

Future Work:

The authors implicitly suggest several directions for future work:

  • Developing specialized, domain-specific languages: To improve productivity and efficiency in writing concurrent programs.
  • Creating generic platforms and infrastructures: For the execution and management of large-scale scientific applications, emphasizing scalability and performance.
  • Integrating HPC seamlessly: Finding the best approaches to enable scientists to leverage HPC technologies transparently and efficiently.
  • Exposing parallelism: Developing languages and systems that expose parallelism explicitly or implicitly, allowing for the scheduling of massive numbers of independent tasks.
  • Establishing common infrastructures: Creating standardized frameworks for workflow administration, scheduling, execution, and provenance tracking, similar to business process management.
  • Separating concerns: Designing structured infrastructures that separate workflow specification, scheduling, and execution, built on specialized components.

Summary updated

Abstract

With the advances in e-sciences and the growing complexity of scientific analyses, more and more scientists and researchers are relying on workflow systems for process coordination, derivation automation, provenance tracking, and bookkeeping. While workflow systems have been in use for decades, it is unclear whether scientific workflows can or even should build on existing workflow technologies, or they require fundamentally new approaches. In this paper, we analyze the status and challenges of scientific workflows, investigate both existing technologies and emerging languages, platforms and systems, and identify the key challenges that must be addressed by workflow systems for e-science in the 21 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">st</sup> century.
BibTeX
@inproceedings{zhao2008scientific,
  author = {Yong Zhao and Ioan Raicu and Ian T. Foster},
  title = {{Scientific Workflow Systems for 21st Century, New Bottle or New Wine?}},
  booktitle = {SERVICES I},
  year = {2008},
  doi = {10.1109/services-1.2008.79},
}
Figure