osely Coupled Programm Petascale Systems

Ioan Raicu, Zhao Zhang, Ian Foster, Pete Beckman, Kamil Iskra, Ben Clifford2008

TL;DR

This paper presents an extension of the Falkon framework to enable efficient execution of "loosely coupled" applications on petascale supercomputers like the IBM Blue Gene/P. They demonstrate that these systems, traditionally optimized for tightly coupled MPI jobs, can effectively run workflows composed of many independent tasks, overcoming challenges like I/O bottlenecks and scheduling overheads to achieve high performance.

What Problem Does This Paper Address?

Modern supercomputers, particularly petascale systems, are increasingly powerful but are primarily designed for tightly coupled parallel applications that use MPI for communication. However, a significant and growing class of scientific applications, termed "Many-Task Computing" (MTC), are inherently "loosely coupled." These applications consist of numerous independent or semi-independent tasks that communicate primarily through files. Traditional HPC systems struggle to efficiently run these MTC applications due to limitations in their local resource managers, file system contention, and scheduling overheads. This gap prevents a broader community of scientists from leveraging the immense power of petascale machines for their workflow-based computations.

Methods

The authors extend the Falkon lightweight task execution framework to address the challenges of running MTC applications on petascale systems. Their approach involves:

  • Multi-level Scheduling: To overcome the coarse granularity of the Blue Gene/P's Cobalt resource manager (which allocates groups of 256 cores), they implement a multi-level scheduler. This allows Falkon to allocate resources from Cobalt at the "pset" level and then manage individual cores within that pset for distinct tasks.
  • Efficient Task Dispatch: They developed a C-based executor and a distributed architecture for Falkon. This replaces the original Java-based, centralized dispatcher with multiple dispatchers running on I/O nodes, significantly improving scalability and reducing load on login nodes.
  • Extensive Caching: To mitigate shared file system (GPFS) bottlenecks, they employ aggressive caching. Application binaries, libraries, common input data, and even dynamic input data are cached in the compute node's RAM disk. This allows tasks to run almost entirely from memory, minimizing slow disk I/O.
  • Swift Integration: They leverage the Swift workflow system, which provides a higher-level scripting and workflow management layer, to orchestrate these loosely coupled tasks.

Datasets

The paper does not explicitly list or introduce new datasets for training. Instead, it uses:

  • Microbenchmarks: Various synthetic tasks (e.g., "sleep" tasks) and file system operations (create directory, create file, script invocation) are used to measure fundamental performance characteristics.
  • Real Applications:
    • DOCK6 and DOCK5: Molecular dynamics applications used for drug discovery, involving large numbers of independent docking simulations.
  • MARS (Macro Analysis of Refinery Systems): An economic modeling application for petroleum refining, used for parameter sweeps and sensitivity analysis.

These applications are chosen to represent typical MTC workloads with varying task execution times and I/O patterns.

Key Insights and Ideas

  • Petascale systems can be efficient for MTC: The core hypothesis is validated – with the right framework and optimizations, petascale machines are not just for MPI jobs but can be powerful platforms for loosely coupled workflows.
  • Shared file systems are the primary bottleneck: The paper repeatedly highlights GPFS contention as the major hurdle for MTC applications on Blue Gene/P. Aggressive caching and intelligent data management are crucial.
  • Task granularity and I/O patterns matter immensely: Short tasks with heavy I/O struggle to achieve high efficiency on distributed systems. Tasks need to be long enough or have minimal I/O to amortize dispatch and I/O overheads.
  • Distributed dispatchers are key for scalability: A single dispatcher becomes a bottleneck at large scales. Distributing the dispatching logic across multiple nodes (like I/O nodes) is essential for handling hundreds of thousands of cores.
  • Loosely coupled applications can achieve high utilization: By effectively managing tasks and minimizing idle time, the authors demonstrate high processor utilization, even on systems designed for tightly coupled workloads.

Key Results

  • Scalability: The Falkon framework, with its distributed architecture and caching, scales efficiently up to 160K processor cores (nearly the full Blue Gene/P system).
  • Task Throughput: Achieved sustained execution rates of thousands of tasks per second, significantly outperforming traditional LRMs like Condor and PBS. For example, 3071 tasks/sec was achieved on 163840 CPUs with distributed dispatchers.
  • Efficiency:
    • Microbenchmarks showed high efficiency (95%+) for short tasks (4 seconds) on smaller scales (up to 2K processors) with a single dispatcher.
  • On larger scales (up to 160K processors) with distributed dispatchers, efficiency for short tasks (1-8 seconds) dropped significantly, requiring task lengths of at least 64 seconds for 90%+ efficiency.
  • Real applications like DOCK6 achieved 95% sustained utilization and 30% overall utilization on 128K cores, with the latter limited by heterogeneous task execution times.
  • MARS application achieved 88% efficiency on 163840 CPUs when using Swift, demonstrating the effectiveness of optimizations for file system access.
  • Startup Costs: Booting the system and initializing Falkon can take significant time (up to 1326 seconds on 160K processors), but this is amortized over long-running applications.
  • File System Performance: GPFS throughput reached 4.4GB/s for reads and 1.3GB/s for read+write with 10MB files on 16K processors. However, file/directory creation costs scaled poorly in a single directory, necessitating splitting I/O across many directories.

Limitations and Future Work

  • Limitations:
    • Efficiency with Short Tasks: Efficiency drops significantly for very short tasks (under 8 seconds) on large-scale distributed systems, especially when using I/O nodes for dispatching.
  • Shared File System Bottleneck: While caching mitigates it, the shared file system remains a fundamental challenge for MTC applications, especially for write-heavy workloads.
  • Data Diffusion Not Fully Utilized: Due to network topology and architecture changes, the full potential of data diffusion techniques was not explored on the Blue Gene/P.
  • Future Work:
    • Leveraging Specialized Networks: Utilize Blue Gene/P's Torus network for more efficient data transfer (e.g., collective I/O operations) to reduce reliance on the shared file system.
  • Data Caching and Replication: Further enhance data caching and explore data-aware scheduling for improved performance.
  • MPI Application Support: Extend Falkon to support running parallel MPI applications as individual tasks within a workflow.
  • Scaling Swift: Continue to scale Swift to larger runs (100K to 1M tasks) and maintain high efficiency at 160K processors.

Summary updated

Abstract

We have extended the Falkon execution framework to make loosely coupled petascale systems a practical and useful prog This work studies and measures the perf involved in applying this approach to enable th systems by a broader user community, and w Our work enables the execution of highly para composed of loosely coupled serial jobs with no the respective applications. This approach all potentially far larger—class of applications to l systems, such as the IBM Blue Gene/P sup present the challenges of I/O performance encou this model practical, and show resul microbenchmarks and real applications from economic energy modeling and molecular benchmarks show that we can scale up to 160K with high efficiency, and can achieve sustained thousands of tasks per second. lightweight task programming on gramming model. formance factors he use of petascale with greater ease. allel computations o modifications to lows a new—and leverage petascale percomputer. We untered in making lts using both m two domains: dynamics. Our K processor-cores execution rates of ughput computing; e; Falkon; Swift
BibTeX
@misc{raicu2008osely,
  author = {Ioan Raicu and Zhao Zhang and Ian Foster and Pete Beckman and Kamil Iskra and Ben Clifford},
  title = {{osely Coupled Programm Petascale Systems}},
  year = {2008},
}

Loading related content...

Figure