A Distributed Re-configurable Grid Workflow Engine

Jian Cao, Minglu Li, Wei Wei, Shen-sheng Zhang

Introduction

In the realm of grid computing, workflows are indispensable for orchestrating complex, large-scale scientific tasks—such as earthquake simulations—across geographically dispersed resources. However, a significant challenge persists: most contemporary workflow engines rely on centralized architectures. As the volume of grid users and concurrent requests grows, these centralized engines inevitably become performance bottlenecks. Furthermore, they represent a single point of failure; an engine slowdown or crash can halt the entire scientific process.

To address these limitations, this research proposes a distributed, re-configurable grid workflow engine designed to eliminate centralized bottlenecks and ensure high availability through dynamic scaling.

System Architecture

The proposed system, the Service-based Grid Workflow Engine (SGWE), adopts a decoupled architecture divided into two primary layers: the Management Layer and the Enactment Layer.

High-level architecture of the Service-based Grid Workflow Engine (SGWE).
  1. Management Layer: Acting as the system's "brain," the Workflow Manager handles user authentication, request processing, and task dispatching. It includes a Service Cache Manager (SCM) for automatic discovery of enactment services and a Load-Balancing Manager (LBM) to optimize task distribution.
  2. Enactment Layer: This layer serves as the system's "muscles," comprising multiple independent services responsible for executing specific workflow tasks, such as model instantiation and task execution.

The SGWE is built upon the Jini platform, a "plug-and-play" networking architecture. This foundation enables the engine's re-configurability: new enactment services can join the network and register themselves via the Jini Discovery Protocol, allowing the Management Layer to utilize them immediately without system restarts.

Dynamic Load Balancing

A core innovation of the SGWE is its approach to load distribution. Rather than simple round-robin assignment, the system employs a weighted load-balancing algorithm. It calculates a "Work Load" (WL) score for each available enactment service using the formula:

$$WL = \sum P_i W_i$$

In this equation, $P_i$ represents performance attributes (e.g., average response time, current queue length), and $W_i$ denotes the weight assigned to each attribute. This mechanism ensures that requests are prioritized for the most efficient and least-burdened services.

Performance Evaluation

The prototype SGWE was evaluated using a synthetic request sequence designed to simulate real-world client access patterns. The evaluation compared a traditional single-service setup against distributed configurations with multiple enactment services.

Average response time vs. number of concurrent users for different service counts.

The results demonstrate significant improvements in scalability. As the number of concurrent users increased from 10 to 140, the distributed SGWE consistently maintained lower average response times compared to the single-service baseline. Notably, the performance advantage of the distributed system became more pronounced as the load intensified. While the single-service response time spiked sharply at 140 users, the 3-service configuration remained relatively stable, proving the effectiveness of the distributed enactment layer.

Conclusion and Future Work

The SGWE prototype successfully demonstrates that a service-based, distributed architecture can effectively mitigate the bottlenecks inherent in centralized workflow engines. By leveraging Jini for dynamic discovery and implementing weighted load balancing, the system achieves high scalability and performance.

Future research will focus on enhancing the load-balancing algorithm to account for the specific computational "cost" of incoming requests. Additionally, the authors aim to develop a "self-deploying" architecture where the engine can autonomously detect load spikes and provision new enactment services on idle grid resources without manual intervention.

Summary updated

Abstract

Grid workflow, as a basic service in the grid environment, is a useful tool to help researchers make use of various gird resources to solve scientific problems. It is possible lots of users invoke grid workflow services in a very narrow time interval. Therefore, a centralized grid workflow engine is apt to be a bottleneck. In the paper, a novel grid workflow engine is proposed. It is based on Jini platform and employs Jini services to implement functions of grid workflow engine. By adding or removing enactment services, the grid workflow engine can be reconfigured dynamically. The workflow manager of the engine can allocate requests to proper enactment services according to some load-balancing strategy. The structure and mechanisms of this grid workflow engine are discussed in the paper. A prototype system and testing results are also introduced.
BibTeX
@inproceedings{cao2006distributed,
  author = {Jian Cao and Minglu Li and Wei Wei and Shen-sheng Zhang},
  title = {{A Distributed Re-configurable Grid Workflow Engine}},
  booktitle = {International Conference on Computational Science},
  year = {2006},
  doi = {10.1007/11758532\_128},
}
Figure