Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices

Yan Sun, Yifan Yuan, Zeduo Yu, Reese Kuper, Chihun Song, Jinghan Huang, Houxiang Ji, Siddharth Agarwal, Jiaqi Lou, Ipoom Jeong, Ren Wang, Jung Ho Ahn, Tianyin Xu, Nam Sung Kim

TL;DR

As Compute Express Link (CXL) memory moves from theory to reality, this paper provides the first comprehensive evaluation of true, commercially available CXL hardware. The authors reveal that common emulation techniques (using remote NUMA nodes) fail to capture real CXL performance characteristics, particularly regarding cache coherence and cache hierarchy interactions. To exploit these findings, they introduce Caption, a dynamic OS page allocation policy that improves the throughput of memory-bandwidth-intensive applications by up to 24%.

What Problem Does This Paper Address?

Modern data centers face a severe "memory wall." Increasing memory capacity and bandwidth using traditional DDR interfaces is hitting physical limits due to CPU pin constraints and high power consumption. Compute Express Link (CXL) has emerged as a open standard to solve this, allowing systems to dynamically expand memory capacity and bandwidth over high-speed serial PCIe links.

Because genuine CXL hardware has been notoriously difficult to acquire, the research community has relied heavily on emulation—specifically, treating a remote CPU socket's memory in a multi-socket system (a remote NUMA node) as a proxy for CXL memory. This paper addresses a critical question: Does emulated CXL memory actually behave like true CXL memory? If not, prior research conclusions and operating system policies might be fundamentally flawed.

Methods

The authors evaluate a true CXL-ready system featuring a 4th-generation Intel Xeon (Sapphire Rapids) processor connected to three distinct, genuine CXL memory devices from different manufacturers:

  • CXL-A: Hard ASIC IP with DDR5-4800 memory.
  • CXL-B: Hard ASIC IP with DDR4-2400 memory.
  • CXL-C: Soft FPGA IP with DDR4-3200 memory.
Figure 1: CXL memory module architecture.

To characterize these devices, they use Intel's Memory Latency Checker (MLC) alongside a custom-built microbenchmark called memo. Unlike standard tools, memo isolates the latency and bandwidth of specific x86 instruction types, such as temporal loads/stores (ld/st) and non-temporal loads/stores (nt-ld/nt-st) which bypass the CPU cache.

Using insights from this hardware characterization, the authors design Caption, an OS-level dynamic page allocation policy. Caption periodically samples hardware performance counters (like L1 miss latency and instructions per cycle) using Intel Performance Counter Monitor (PCM). It feeds these metrics into a lightweight linear regression model to predict system throughput and dynamically adjusts the ratio of pages allocated to local DDR vs. CXL memory using a greedy optimization algorithm.

Datasets

The paper evaluates system and application performance across a diverse suite of workloads:

  • Latency-Sensitive Applications: Redis (benchmarked via the Yahoo! Cloud Serving Benchmark, YCSB), DeathStarBench (an open-source social network microservice framework), and FIO (flexible I/O tester evaluating OS page cache performance).
  • Throughput-Intensive Applications: Facebook’s Deep Learning Recommendation Model (DLRM) embedding reduction step, and the four highest memory-bandwidth-demanding benchmarks from the SPECrate CPU2017 suite (fotonik3d, mcf, roms, and cactuBSSN).

Key Insights and Ideas

  • CXL Memory $\neq$ Remote NUMA Memory: True CXL memory lacks remote CPU cores and caches. Consequently, the local CPU utilizes an on-chip hardware structure for fast cache coherence checks rather than traversing slow inter-socket interconnects (like Intel UPI). This allows true CXL to achieve up to 26% lower latency and significantly higher bandwidth efficiency for parallel memory accesses compared to emulated setups.
  • Breaking LLC Isolation: In Sub-NUMA Clustering (SNC) mode, the CPU normally isolates Last Level Cache (LLC) slices to prevent applications from interfering with one another. However, the authors discovered that accessing CXL memory breaks this isolation: evicted L2 cache lines from CXL memory can be distributed across all LLC slices in any SNC node. This effectively grants CXL-accessing applications a 2–4× larger LLC capacity, mitigating CXL's latency penalties for cache-friendly workloads.
Figure 5: Difference in L2 cache line eviction paths between local DDR memory and CXL memory in SNC mode.
  • Naïve Allocation is Harmful: Blindly splitting pages 50/50 between local DDR and CXL memory (the default OS behavior) degrades performance. Microsecond-scale applications like Redis experience tail-latency spikes of 10–82%. Even bandwidth-hungry applications can suffer lower throughput due to memory controller contention if allocations aren't dynamically balanced based on the specific CXL device's capabilities.

Key Results

  • Latency Variations: Hardware controller design matters immensely. The ASIC-based CXL-A device exhibited only 35% longer load latency than local DDR5, whereas the FPGA-based CXL-C device suffered nearly 3× longer latency (see Figure 3).
Figure 3: Random memory access latency of various memory devices (DDR5-R, CXL-A, CXL-B, and CXL-C), measured by Intel MLC and memo.
  • Caption Policy Performance: When evaluating mixed throughput workloads (SPEC-Mix), Caption successfully navigated the trade-offs between DDR and CXL bandwidth. It automatically converged on optimal allocation ratios (allocating 29%–41% of pages to CXL), outperforming the best static OS allocation policies by up to 24%.
  • Page Cache Viability: Allocating the Linux page cache to CXL memory using FIO resulted in a mere ~3% latency increase for 4 KB blocks, proving that CXL is an excellent, cost-effective candidate for OS caching layers.

Limitations and Future Work

While Caption demonstrates excellent results, its predictive linear regression model was primarily trained on weight values derived from DLRM profiles. The authors note that accuracy could be further improved by training the model on a wider, more diverse set of applications. Additionally, current hardware limitations prevent the OS from directly reading performance counters located on the CXL devices themselves (such as device-side queue lengths or exact link utilization). Accessing these device-native statistics remains an open avenue for building even more responsive tiered-memory managers.

Summary updated

Abstract

The ever-growing demands for memory with larger capacity and higher bandwidth have driven recent innovations on memory expansion and disaggregation technologies based on Compute eXpress Link (CXL). Especially, CXL-based memory expansion technology has recently gained notable attention for its ability not only to economically expand memory capacity and bandwidth but also to decouple memory technologies from a specific memory interface of the CPU. However, since CXL memory devices have not been widely available, they have been emulated using DDR memory in a remote NUMA node. In this paper, for the first time, we comprehensively evaluate a true CXL-ready system based on the latest 4th-generation Intel Xeon CPU with three CXL memory devices from different manufacturers. Specifically, we run a set of microbenchmarks not only to compare the performance of true CXL memory with that of emulated CXL memory but also to analyze the complex interplay between the CPU and CXL memory in depth. This reveals important differences between emulated CXL memory and true CXL memory, some of which will compel researchers to revisit the analyses and proposals from recent work. Next, we identify opportunities for memory-bandwidth-intensive applications to benefit from the use of CXL memory. Lastly, we propose a CXL-memory-aware dynamic page allocation policy, Caption to more efficiently use CXL memory as a bandwidth expander. We demonstrate that Caption can automatically converge to an empirically favorable percentage of pages allocated to CXL memory, which improves the performance of memory-bandwidth-intensive applications by up to 24% when compared to the default page allocation policy designed for traditional NUMA systems.
BibTeX
@inproceedings{sun2023demystifying,
  author = {Yan Sun and Yifan Yuan and Zeduo Yu and Reese Kuper and Chihun Song and Jinghan Huang and Houxiang Ji and Siddharth Agarwal and Jiaqi Lou and Ipoom Jeong and Ren Wang and Jung Ho Ahn and Tianyin Xu and Nam Sung Kim},
  title = {{Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices}},
  booktitle = {MICRO},
  year = {2023},
  doi = {10.1145/3613424.3614256},
}
Figure