Abstract
UniFaaS is a Python-based framework that enables researchers to run complex scientific workflows across a "federated" landscape of distributed supercomputers, clouds, and local clusters. By treating tasks as lightweight functions (FaaS) and employing a smart, heterogeneity-aware scheduler, it can speed up real-world scientific discovery by over 50% compared to using a single computing site.
Introduction
Modern scientific research—such as drug discovery or astronomical imaging—increasingly relies on "fine-grained" workflows. These applications are composed of thousands of small, individual tasks (functions) with diverse requirements: some need GPUs, others massive RAM, and many must be located near specific datasets. While researchers often have access to multiple computing resources (e.g., institutional clusters, national supercomputers, and public clouds), utilizing them in concert is challenging. Traditional batch schedulers suffer from long wait times, and existing parallel frameworks like Ray or Dask are not designed to span different administrative domains or wide-area networks. This leads to manual data management and underutilized hardware.
System Architecture
UniFaaS addresses these challenges by leveraging a federated Function-as-a-Service (FaaS) model built on top of funcX (now known as Globus Compute). The architecture consists of a unified programming interface, an observe-predict-decide loop, and a transparent data management layer.
The system uses a Python decorator (@function) and RemoteFile objects to allow developers to write code that executes globally while appearing local. It monitors execution in real-time and employs Random Forest regression to predict task durations on specific hardware endpoints.
Dynamic Heterogeneity-Aware (DHA) Scheduling
The core innovation of UniFaaS is the Dynamic Heterogeneity-Aware (DHA) scheduling algorithm. DHA is a hybrid approach that prioritizes tasks based on predicted execution and data transfer times.
DHA incorporates several key mechanisms:
- Priority-Based Scheduling: Inspired by the HEFT algorithm, it ranks tasks by their impact on total makespan.
- Delay Mechanism: It may hold a task in a queue to wait for a high-performance resource (e.g., a supercomputer) rather than immediately dispatching it to a slower, available one.
- Dynamic Re-scheduling: If resource availability changes, the system can "steal" pending tasks from other queues to optimize performance.
Evaluation and Key Results
The system was evaluated using two large-scale scientific workflows: a Drug Screening pipeline (24,001 functions) and the Montage astronomical image mosaic engine (11,340 functions). Experiments were conducted on a diverse testbed including the Taiyi supercomputer, the Qiming cluster, and various departmental clusters.
UniFaaS demonstrated significant performance improvements:
- Speedups: The Montage workflow saw a 54.41% improvement by utilizing 47.83% more resources across multiple clusters. The drug screening workflow achieved a 22.99% boost.
- Scheduling Efficiency: The DHA algorithm outperformed standard locality-aware or capacity-aware schedulers by up to 32% in dynamic environments where resource availability fluctuated.
- Overhead: The framework adds only millisecond-level latency to scheduling and profiling, making it suitable for short-lived tasks.
A key insight is that the "Transfer Tax"—the cost of moving data between sites—can be effectively hidden by overlapping data staging with computation. Furthermore, the DHA scheduler proves that in heterogeneous environments, waiting for a superior resource is often faster than immediate execution on a slower one.
Conclusion and Future Work
UniFaaS provides a scalable, elastic "glue" for distributed cyberinfrastructure, removing the friction of manual job scripts and environment mismatches. Future work will focus on more sophisticated network-aware data profiling and advanced coordination strategies for multi-endpoint elasticity to prevent bottlenecks during scaling.
Summary updated