A resource management architecture for metacomputing systems.

Karl Czajkowski, Ian Foster, Nicholas T. Karonis, Carl Kesselman, Steven Martin, Warren Smith, Steven Tuecke

TL;DR

This paper introduces a novel resource management architecture for metacomputing systems, designed to handle the complexities of distributed, heterogeneous computing environments. It addresses key challenges like site autonomy, policy extensibility, and co-allocation through a layered approach involving resource brokers, co-allocators, and local resource managers, all communicating via an extensible resource specification language.

What Problem Does This Paper Address?

The paper tackles the significant challenge of resource management in "metacomputing" systems. These systems aim to let users seamlessly access and utilize geographically distributed computational resources as if they were a single, powerful machine. However, this vision is hindered by several practical issues that don't typically arise in single-site computing:

  • Site Autonomy: Different organizations own and manage resources, each with its own policies for usage, security, and scheduling. A metacomputing system must respect these independent policies.
  • Heterogeneous Substrates: The underlying computing resources at each site can vary wildly, using different local resource management systems (like Condor, LSF, PBS, etc.) with unique configurations and capabilities.
  • Policy Extensibility: Metacomputing applications come from diverse domains, each with unique resource needs and priorities. The resource management system must be flexible enough to accommodate these varied and evolving requirements.
  • Co-allocation: Many advanced applications require resources to be allocated simultaneously across multiple sites to function correctly. Coordinating this across independent systems is complex.
  • Online Control: Applications might need to adapt their resource demands dynamically during execution (e.g., requesting more or fewer resources based on real-time performance). The system must support this negotiation and adaptation.

Existing systems often address only a subset of these problems, leaving a gap for a comprehensive solution.

Methods

The authors propose a layered architecture that breaks down the resource management problem:

  • Extensible Resource Specification Language (RSL): This is the core communication mechanism. RSL allows applications to express their resource needs in a flexible, extensible way. It supports complex queries, including conjunctions and disjunctions of requirements, and can be specialized by different components.
  • Resource Brokers: These act as intermediaries. They take high-level application requests (expressed in RSL) and "specialize" them into more concrete requirements. This specialization can involve translating application-specific needs (e.g., "high throughput for simulations") into more technical specifications (e.g., "X GFLOPS, Y GB RAM"). Multiple brokers can be chained together to refine a request.
  • Resource Co-allocators: For applications requiring simultaneous allocation of resources across multiple sites, co-allocators handle the complex task of coordinating these requests. They break down a multirequest into individual requests for resource managers and manage the collective set of allocated resources.
  • Local Resource Managers (GRAMs): These are the lowest-level components, acting as interfaces to the local resource management systems at each site (e.g., LSF, Condor). GRAMs translate the specialized RSL requests into commands understood by the local scheduler and report back resource status. They also handle authentication and process creation.
  • Information Service (MDS): A crucial component that provides up-to-date information about the status, capabilities, and availability of resources across the metacomputing environment. This information is vital for brokers and co-allocators to make informed decisions.

This architecture distributes the complexity, allowing specialized components to handle specific challenges.

Datasets

Not discussed in the paper. The paper focuses on the architecture and its implementation on a testbed.

Key Insights and Ideas

  • Layered Abstraction is Key: The most significant insight is the decomposition of the complex metacomputing resource management problem into distinct, manageable layers (brokers, co-allocators, GRAMs). This modularity allows for flexibility and extensibility.
  • RSL as a Universal Language: The development of a powerful and extensible RSL is critical. It acts as a common ground for communication between diverse components and systems, enabling negotiation and dynamic adaptation.
  • Separation of Concerns: The architecture clearly separates the concerns of high-level policy and application requirements (handled by brokers) from the low-level execution and local resource management (handled by GRAMs).
  • The Power of a Centralized Information Service: The reliance on an information service (MDS) to provide a global view of resource availability is essential for efficient and intelligent resource allocation.
  • Practicality through Existing Tools: The architecture is designed to integrate with and leverage existing local resource management systems, rather than requiring their replacement, making it more practical to adopt.

Key Results

The paper reports on the implementation and deployment of this architecture within the Globus metacomputing toolkit. A key validation effort was conducted on the GUSTO testbed, which comprised:

  • 15 sites
  • 330 computers
  • 3600 processors

The system was tested with various local schedulers, including LSF, NQE, LoadLeveler, EASY, Fork, and Condor. The authors demonstrate that their architecture successfully addresses the five core challenges (site autonomy, heterogeneity, policy extensibility, co-allocation, and online control). They also highlight the successful use of the architecture to manage resources for the SF-Express application, simultaneously allocating 852 compute nodes across three different architectures at six different computer centers.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Scalability of Co-allocation: The "all or nothing" semantics of their current co-allocation strategy can limit scalability, especially with a large number of subjobs. Failures in one part can lead to the cancellation of the entire allocation.
  • Dynamic Job Structure Modification: They propose extending co-allocation to support dynamic modification of job structures, allowing for substitutions of failed resources rather than complete cancellation. This would involve brokers editing specifications based on failure information.
  • More Sophisticated Brokers and Co-allocators: Future work will focus on developing more advanced resource broker and co-allocator services.
  • Extending to Other Resources: The architecture could be extended to manage other resources like disk and network.
  • Automatic Policy Negotiation: Investigating how policy information can be encoded to facilitate automatic negotiation between resources, users, and intermediaries is a key area for future research.

Summary updated

Abstract

Metacomputing systems are intended to support remote and/or concurrent use of geographically distributed computational resources. Resource management in such systems is complicated by five concerns that do not typically arise in other situations: site autonomy and heterogeneous substrates at the resources, and application requirements for policy extensibility, co-allocation, and online control. We describe a resource management architecture that addresses these concerns. This architecture distributes the resource management problem among distinct local manager, resource broker, and resource co-allocator components and defines an extensible resource specification language to exchange information about requirements. We describe how these techniques have been implemented in the context of the Globus metacomputing toolkit and used to implement a variety of different resource management strategies. We report on our experiences applying our techniques in a large testbed, GUSTO, incorporating 15 sites, 330 computers, and 3600 processors.
BibTeX
@inproceedings{czajkowski1999resource,
  author = {Karl Czajkowski and Ian Foster and Nicholas T. Karonis and Carl Kesselman and Steven Martin and Warren Smith and Steven Tuecke},
  title = {{A resource management architecture for metacomputing systems.}},
  booktitle = {University},
  year = {1999},
}
Figure