CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads

Radostin Stoyanov, Viktória Spišaková, J.i. Ramos, Steven Gurfinkel, Andrei Vagin, Adrian Reber, Wesley Armour, Rodrigo Bruno

Abstract

Deep learning training at scale is resource-intensive and time-consuming, often running across hundreds or thousands of GPUs for weeks or months. Efficient checkpointing is crucial for running these workloads, especially in multi-tenant environments where compute resources are shared, and job preemptions or interruptions are common. However, transparent and unified GPU snapshots are particularly challenging because of the hardware architecture differences between CPU and GPU, including memory subsystems, dynamic parallelism, and thread synchronization. State-of-the-art GPU checkpointing techniques typically leverage mechanisms that intercept, log, and replay device API calls. However, this approach adds performance overhead and requires hardware-specific implementation that is difficult to test, maintain, and integrate with existing container platforms. In this paper, we present CRIUgpu - a novel approach for transparent checkpointing of GPU-accelerated workloads that builds on recently introduced driver capabilities, enabling support for CUDA and ROCm applications. Our evaluation results show that CRIUgpu works with a variety of deep learning and high-performance computing workloads running across multiple GPUs, completely eliminating steady-state performance overheads, and significantly reducing recovery times compared to state-of-the-art transparent GPU checkpointing mechanisms.

Introduction

Training modern large language models (LLMs) like LLaMA 3.1 is a massive undertaking, often requiring 16,000 GPUs to run for several months. In such large-scale deployments, hardware interruptions are a statistical certainty. To prevent the loss of weeks of computational progress, researchers rely on checkpointing—periodically saving the system state to disk.

The ideal solution is transparent checkpointing, which captures the state of a running application without requiring any modifications to its source code. However, implementing this for GPUs has historically been difficult. Most existing solutions rely on "API interception" or device proxies. As illustrated in Figure 1, these proxies sit between the application and the GPU driver, logging every command to reconstruct the state later. This method introduces significant "steady-state" performance overhead, struggles with modern features like static linking (common in PyTorch), and requires constant maintenance to keep up with driver updates.

Comparison of CRIUgpu's native plugin architecture (a) versus traditional API-interception device proxies (b).

The CRIUgpu Approach

CRIUgpu addresses these challenges by moving away from API interception. Instead, it leverages native checkpointing capabilities recently introduced by GPU vendors. Built as an extension to CRIU (Checkpoint/Restore in Userspace), the industry-standard tool for Linux container snapshots, CRIUgpu uses specialized plugins to interface directly with GPU drivers.

  • NVIDIA Support: The system utilizes the cuda-checkpoint utility. When a snapshot is initiated, the plugin locks GPU APIs, waits for active kernels to complete, and instructs the driver to dump the internal GPU state (including memory, streams, and contexts) into host memory.
  • AMD Support: The system interfaces with the Kernel Fusion Driver (KFD) via ioctl operations. It pauses hardware queues and captures "Memory Buffer Objects" (BOs) that define the GPU's state.

By coordinating these actions with the standard CPU checkpointing process, CRIUgpu creates a single, consistent image of the entire containerized environment.

Performance and Evaluation

The primary advantage of CRIUgpu is the elimination of runtime performance penalties. Because it does not intercept API calls during normal execution, applications run at 100% native speed. In contrast, interception-based tools like Cricket exhibit substantial overhead that scales with the number of API calls and data samples, as shown in Figure 2.

Steady-state performance overhead of interception-based tools (Cricket) compared to the native baseline. CRIUgpu eliminates this overhead.

In empirical tests across various hardware (NVIDIA H100, A100, and AMD MI210), CRIUgpu demonstrated impressive efficiency:

  • LLaMA 3.1 (8B): On an NVIDIA H100, a full checkpoint was created in 77.4 seconds and restored in 38.8 seconds.
  • Compatibility: Unlike proxy-based methods, CRIUgpu works seamlessly with statically linked binaries, supporting standard installations of PyTorch and TensorFlow.
  • Scalability: The system scales near-linearly with the number of GPUs in a single node, handling the increased VRAM state efficiently.

Discussion and Future Work

While CRIUgpu represents a significant advancement, some challenges remain. Currently, the system is optimized for single-node, multi-GPU configurations. Extending this to multi-node setups involves managing high-speed fabrics like InfiniBand and libraries like NCCL, which do not yet have full native checkpointing support. Additionally, as model sizes grow, the resulting checkpoint files (often exceeding 60GB) present storage and I/O bottlenecks. Future research will explore incremental checkpointing and data compression to mitigate these issues.

CRIUgpu has been integrated into the upstream CRIU project (v4.0) and is already seeing use in production environments, providing a robust foundation for resilient, large-scale AI infrastructure.

Summary updated

Abstract

Deep learning training at scale is resource-intensive and time-consuming, often running across hundreds or thousands of GPUs for weeks or months. Efficient checkpointing is crucial for running these workloads, especially in multi-tenant environments where compute resources are shared, and job preemptions or interruptions are common. However, transparent and unified GPU snapshots are particularly challenging because of the hardware architecture differences between CPU and GPU, including memory subsystems, dynamic parallelism, and thread synchronization. State-of-the-art GPU checkpointing techniques typically leverage mechanisms that intercept, log, and replay device API calls. However, this approach adds performance overhead and requires hardware-specific implementation that is difficult to test, maintain, and integrate with existing container platforms. In this paper, we present CRIUgpu - a novel approach for transparent checkpointing of GPU-accelerated workloads that builds on recently introduced driver capabilities, enabling support for CUDA and ROCm applications. Our evaluation results show that CRIUgpu works with a variety of deep learning and high-performance computing workloads running across multiple GPUs, completely eliminating steady-state performance overheads, and significantly reducing recovery times compared to state-of-the-art transparent GPU checkpointing mechanisms.
BibTeX
@misc{stoyanov2025criugpu,
  author = {Radostin Stoyanov and Viktória Spišaková and J.I. Ramos and Steven Gurfinkel and Andrei Vagin and Adrian Reber and Wesley Armour and Rodrigo Bruno},
  title = {{CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads}},
  year = {2025},
  eprint = {2502.16631},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.16631},
}

Loading related content...

Figure