Assessing the Use Cases of Persistent Memory in High-Performance Scientific Computing

Yehonatan Fridman, Yaniv Snir, Matan Rusanovsky, Kfir Zvi, Harel Levin, Danny Hendler, Hagit Attiya, Gal Oren

Introduction

In the pursuit of Exascale computing, traditional high-performance computing (HPC) architectures face two critical bottlenecks: the "memory wall" and the "resilience bottleneck." The memory wall refers to the massive performance gap between volatile DRAM and non-volatile storage. When scientific simulations exceed a node's DRAM capacity, they must rely on disk-based swapping, which typically results in a performance collapse of several orders of magnitude. Simultaneously, the resilience bottleneck arises from the need for frequent "Checkpoint/Restart" (C/R) operations to safeguard against system failures. Writing massive state files to slow storage consumes a disproportionate amount of total computation time.

This study evaluates Intel Optane Persistent Memory (DCPMM) as a potential "middle tier" to bridge this gap, offering near-DRAM speeds with non-volatility.

Experimental Methodology

The evaluation was conducted on a dual-socket server featuring 192GB of DDR4 DRAM and 1TB of Intel Optane DCPMM. Two primary operational modes were tested:

  1. Memory Mode: DCPMM acts as the main volatile memory pool, with DRAM serving as a transparent cache. This mode requires no code modifications.
  2. App Direct Mode: DCPMM is treated as a byte-addressable persistent storage device, accessed via specialized persistent memory file systems (PMFS) such as NOVA and SplitFS, or standard filesystems like ext4-dax.

Workloads included the NAS Parallel Benchmarks (NPB) for general scientific kernels, PolyBench for fine-grained memory scaling, and I/O benchmarks like FIO and BTIO.

Performance as Memory Expansion: The "Soft Landing"

A key finding of this research is the dramatic difference in performance degradation when applications exceed DRAM capacity. Traditional swapping to a SATA-SSD causes throughput to plummet by 100x to 1000x as soon as the DRAM limit is reached. In contrast, using DCPMM in Memory Mode provides a "soft landing," where performance only drops by approximately 2x to 3x.

Floating-point throughput comparison between DCPMM expansion and SATA-SSD swap.

As shown in the figure above, the NPB and PolyBench results demonstrate that DCPMM allows researchers to run significantly larger problem sizes on a single node without the catastrophic performance loss associated with traditional disk swapping.

Persistent Storage and Checkpointing Performance

In App Direct mode, DCPMM serves as a high-performance target for Checkpoint/Restart operations. The study compared explicit checkpointing (using the SCR library) and transparent checkpointing (using DMTCP). While transparent methods are easier to implement, they produce larger files. However, the high bandwidth of DCPMM mitigates this overhead.

I/O bandwidth comparison across different filesystems on PM-Optane vs SATA-SSD.

The I/O benchmarks reveal that DCPMM achieves up to 10x faster reads and 3.5x faster writes compared to SATA-SSDs. Furthermore, the choice of filesystem is critical; specialized PMFS like NOVA and SplitFS significantly outperform standard Linux filesystems in write-heavy scenarios. Overall, using DCPMM for C/R was found to be 1.12x to 2.62x faster than using an SSD.

Conclusion

Persistent memory represents a transformative technology for HPC, providing a viable solution to both memory capacity constraints and I/O bottlenecks. By offering a 2-3 order of magnitude speedup over traditional SSD swapping and drastically reducing checkpointing overhead, DCPMM enables more efficient and resilient large-scale simulations. While the specific Optane hardware is transitioning to legacy status, the architectural principles of tiered memory and high-speed persistent tiers—now evolving into CXL-based solutions—remain fundamental to the future of Exascale computing.

Summary updated

BibTeX
@inproceedings{fridman2021assessing,
  author = {Yehonatan Fridman and Yaniv Snir and Matan Rusanovsky and Kfir Zvi and Harel Levin and Danny Hendler and Hagit Attiya and Gal Oren},
  title = {{Assessing the Use Cases of Persistent Memory in High-Performance Scientific Computing}},
  booktitle = {FTXS@SC},
  year = {2021},
  doi = {10.1109/ftxs54580.2021.00006},
}
Figure