Fork is All You Need in Heterogeneous Systems

Zixuan Wang, Jishen ZhaoarXiv 2024 · OpenReview

Abstract

We present a unified programming model for heterogeneous computing systems. Such systems integrate multiple computing accelerators and memory units to deliver higher performance than CPU-centric systems. Although heterogeneous systems have been adopted by modern workloads such as machine learning, programming remains a critical limiting factor. Conventional heterogeneous programming techniques either impose heavy modifications to the code base or require rewriting the program in a different language. Such programming complexity stems from the lack of a unified abstraction layer for computing and data exchange, which forces each programming model to define its abstractions. However, with the emerging cache-coherent interconnections such as Compute Express Link (CXL), we see an opportunity to standardize such architecture heterogeneity and provide a unified programming model. We present CodeFlow, a language runtime system for heterogeneous computing. CodeFlow abstracts architecture computation in programming language runtime and utilizes CXL as a unified data exchange protocol. Workloads written in high-level languages such as C++ and Rust can be compiled to CodeFlow, which schedules different parts of the workload to suitable accelerators without requiring the developer to implement code or call APIs for specific accelerators. CodeFlow reduces programmers' effort in utilizing heterogeneous systems and improves workload performance.

The Challenge of Heterogeneous Programming

Modern computing is increasingly "heterogeneous," relying on a mix of CPUs, GPUs, SmartNICs, and FPGAs to handle specialized workloads like AI and graph processing. However, programming these systems is currently a fragmented nightmare. Developers typically face three major hurdles: rewriting code in specialized languages like CUDA or OpenCL, manually managing data by explicitly copying memory between devices over slow PCIe buses, and handling complex toolchains for every new piece of hardware.

Fragmented programming models in current heterogeneous systems require specialized libraries and drivers for each hardware component.

This complexity stems from a lack of a unified abstraction layer. While multi-core CPU programming is intuitive thanks to shared memory and standard "fork/join" models, heterogeneous systems have historically lacked the shared "language" and cache coherence needed to make this possible.

The CodeFlow Framework

CodeFlow bridges the gap between high-level software and diverse hardware by combining two key technologies into a unified runtime system.

  1. WebAssembly System Interface (WASI): CodeFlow uses WASI as a universal intermediate representation. Developers write code in high-level languages (C++, Rust), which is compiled into a single WASI binary. The CodeFlow runtime then performs Just-In-Time (JIT) or Ahead-Of-Time (AOT) compilation to translate that binary into the specific machine code required by the target accelerator.
  2. Compute Express Link (CXL): This hardware interconnect protocol provides cache coherence across different devices. This allows a GPU and a CPU to access the same memory address without the programmer needing to manually manage data transfers via memcpy.
The CodeFlow pipeline: from high-level language to WASI binary, through the JIT runtime, to execution on a heterogeneous system.

When a program spawns a thread, the CodeFlow runtime analyzes the task and handles scheduling transparently—for instance, sending file-system tasks to the CPU while offloading heavy data processing to a GPU or FPGA.

Key Insights and Performance

The core innovation of CodeFlow is treating heterogeneity as a form of multi-threading. By solving the memory coherence problem (via CXL) and the instruction set problem (via WASI), an accelerator effectively becomes just another "core." This allows programmers to use the familiar fork model to offload work, decoupling high-level logic from specific hardware targets.

Evaluation of the system showed that this abstraction comes with a "zero-tax" performance profile. CodeFlow (WASI) demonstrated nearly identical memory access latency to native x86 code across most memory access sizes. While Local CXL latency was measured at 371.2 ns—higher than local DDR5 (108.2 ns)—it remains significantly more efficient than traditional non-coherent methods for cross-device communication. Most importantly, the authors demonstrated a workflow where a single C++ program is compiled once and distributed across different hardware units without any hardware-specific code modifications.

Future Directions

While promising, the current implementation of CodeFlow faces some limitations. Evaluation was primarily limited to FPGAs as CXL-capable GPUs and SmartNICs are still emerging. Future work will focus on more sophisticated profiling-guided optimizations to automatically place data in the most efficient memory tiers and expanding compiler support to a wider array of specialized accelerators.

Summary updated

Abstract

We present a unified programming model for heterogeneous computing systems. Such systems integrate multiple computing accelerators and memory units to deliver higher performance than CPU-centric systems. Although heterogeneous systems have been adopted by modern workloads such as machine learning, programming remains a critical limiting factor. Conventional heterogeneous programming techniques either impose heavy modifications to the code base or require rewriting the program in a different language. Such programming complexity stems from the lack of a unified abstraction layer for computing and data exchange, which forces each programming model to define its abstractions. However, with the emerging cache-coherent interconnections such as Compute Express Link, we see an opportunity to standardize such architecture heterogeneity and provide a unified programming model. We present CodeFlow, a language runtime system for heterogeneous computing. CodeFlow abstracts architecture computation in programming language runtime and utilizes CXL as a unified data exchange protocol. Workloads written in high-level languages such as C++ and Rust can be compiled to CodeFlow, which schedules different parts of the workload to suitable accelerators without requiring the developer to implement code or call APIs for specific accelerators. CodeFlow reduces programmers' effort in utilizing heterogeneous systems and improves workload performance.
BibTeX
@misc{wang2024fork,
  author = {Zixuan Wang and Jishen Zhao},
  title = {{Fork is All You Need in Heterogeneous Systems}},
  year = {2024},
  eprint = {2404.05085},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05085},
}
Figure