Abstract
The FFMK project designs, builds and evaluates a system-software architecture to address the challenges expected in Exascale systems. In particular, these challenges include performance losses caused by the much larger impact of runtime variability within applications, hardware, and operating system (OS), as well as increased vulnerability to failures. The FFMK OS platform is built upon a multi-kernel architecture, which combines the L4Re microkernel and a virtualized Linux kernel into a noise-free, yet feature-rich execution environment. It further includes global, distributed platform management and system-level optimization services that transparently minimize checkpoint/restart overhead for applications. The project also researched algorithms to make collective operations fault tolerant in presence of failing nodes. In this paper, we describe the basic components, algorithms, and services we developed in Phase 2 of the project.
Introduction: The Exascale Challenge
As high-performance computing (HPC) pushes toward exascale systems capable of $10^{18}$ calculations per second, two primary hurdles emerge: OS noise and fault tolerance.
In massive parallel systems, applications typically follow a Bulk-Synchronous Parallel (BSP) model. In this model, a single interruption—such as a background daemon or timer interrupt on one node—can stall the entire system, leading to significant performance degradation. This "jitter" or OS noise becomes a dominant factor as systems scale to millions of cores.
Furthermore, the sheer number of hardware components in an exascale system drastically reduces the mean time between failures (MTBF). Traditional checkpoint/restart (C/R) methods, where the entire system halts to save state, become unsustainable bottlenecks if they are uncoordinated or too frequent.
Architecture and Methods
To address these issues, the FFMK project developed a multi-kernel architecture that decouples application execution from standard OS services.
The Multi-Kernel Stack
- L4Re Microkernel: This minimal kernel provides a high-performance, "noise-free" environment for compute-intensive tasks.
- L4Linux: A paravirtualized Linux kernel runs atop L4Re, providing the standard APIs, libraries, and drivers (e.g., InfiniBand) required by HPC applications.
- Thread Decoupling: Application threads are extracted from the Linux scheduler to run directly on dedicated "compute cores" managed by the microkernel. Non-critical system calls are transparently offloaded to the Linux kernel running on separate "service cores."
Fault-Tolerant Services
Beyond the kernel architecture, the project introduced several key services:
- Fault-Tolerant Collective Operations (FTCO): Tree-based MPI algorithms (like
Allreduce) use per-node timeouts to bypass failed nodes in real-time, preventing application crashes. - Diffusion-Based Load Balancing: A nearest-neighbor repartitioning method balances workloads across nodes with minimal process migration and low computational overhead.
- Coordinated Checkpointing: By coordinating checkpoints across multiple parallel jobs sharing the same I/O infrastructure, the system reduces overhead by up to 20%.
Evaluation and Results
The FFMK components were evaluated using standard HPC benchmarks (FWQ, HPCCG, DGEMM) and production clusters like Taurus (TU Dresden) and Piz Daint (ETH Zurich).
- Jitter Reduction: Decoupling reduced execution-time jitter from 55 CPU cycles per iteration to just 4 cycles in lab settings.
- Scaling: In a BSP-style run across 750 cores, FFMK's decoupled threads outperformed standard Linux-scheduled threads by approximately 1%.
- Fault Tolerance Efficiency: The FTCO
Allreducelatency was found to be proportional only to the number of serial faults; parallel faults in different sub-trees did not increase recovery time. - Load Balancing Speed: The diffusion-based algorithm proved to be 1–2 orders of magnitude faster than traditional graph partitioners like ParMetis, completing in 2.5 ms for 6,144 processes.
Insights and Future Work
A significant finding of the project is that hardware itself introduces variability; even in a perfectly quiet software environment, microarchitectural differences can cause performance variations across six orders of magnitude. By using a microkernel as a "Universal Lightweight Kernel (LWK)," FFMK avoids the maintenance burden of constantly patching mainline Linux to reduce noise.
While individual building blocks are functional, future work focuses on the full integration of process-level migration with InfiniBand hardware and further refining the SEMPEROS capability system for extreme-scale parallelism.
Summary updated