On the Applicability of PEBS based Online Memory Access Tracking for Heterogeneous Memory Management at Scale

Aleix Roca Nonell, Balazs Gerofi, Leonardo Bautista-Gomez, Dominique Martinet, Vicenç Beltran Querol, Yutaka Ishikawa

TL;DR

This paper investigates the feasibility of using Intel's Processor Event-Based Sampling (PEBS) to track memory access patterns in real-time for heterogeneous memory management on large-scale High-Performance Computing (HPC) systems. The authors developed a custom PEBS driver in a lightweight OS and found that it can capture access patterns with a low overhead (1-10%), making it a promising approach for future memory management strategies.

What Problem Does This Paper Address?

The advent of new memory technologies like High-Bandwidth Memory (HBM) and Non-Volatile Storage Class Memories (NVM) is leading to heterogeneous memory systems. Effectively managing these diverse memory types requires understanding how applications access data to optimize placement and movement. Traditional methods for tracking memory access patterns often rely on dynamic instrumentation, which incurs prohibitive overhead, especially for large-scale parallel applications. This leads to a two-phase approach: first profiling, then applying a policy. This paper explores whether a single-phase, online approach using hardware-assisted sampling (PEBS) can provide the necessary information with acceptable overhead on massive HPC systems.

Methods

The core of this work is the development of a custom PEBS driver within the IHK/McKernel lightweight multi-kernel operating system. This LWK was chosen for its minimal system interference, enabling cleaner characterization of PEBS overhead. The custom driver offers fine-grained control over PEBS parameters, such as the "reset counter value" (which dictates sampling frequency) and the PEBS buffer size, which are not as easily tunable in standard Linux drivers. The driver captures load instruction addresses that trigger L2 cache misses (as this event is supported by PEBS on the target architecture, Intel Xeon Phi Knights Landing). This data is then processed to reconstruct memory access patterns. The evaluation involves running several scientific applications on up to 128,000 CPU cores and measuring the performance overhead introduced by the PEBS tracking.

Datasets

The paper evaluates its approach using a set of established HPC mini-applications and benchmarks:

  • GeoFEM: A 3D linear elasticity solver.
  • HPCG: High Performance Conjugate Gradients benchmark.
  • LAMMPS: A classical molecular dynamics simulator.
  • MiniFE: A proxy application for unstructured implicit finite element codes.
  • Lulesh: A Livermore Unstructured Lagrangian Explicit Shock Hydrodynamics code.
  • AMG2013: A parallel algebraic multigrid solver.

These benchmarks are representative of scientific workloads and were run in various configurations (weak and strong scaling) to assess the PEBS overhead across different scales and application behaviors.

Key Insights and Ideas

  • PEBS is viable for large-scale HPC: Contrary to some previous findings, this work demonstrates that PEBS can be effectively used for online memory access tracking even on massive HPC systems (up to 128k cores) without prohibitive overhead.
  • Fine-grained control matters: The custom driver's ability to tune PEBS parameters (reset value and buffer size) is crucial. Lowering the reset value (increasing sampling frequency) leads to more detailed access patterns, and the overhead can be managed by adjusting these parameters.
  • Overhead is application-dependent: The performance impact of PEBS tracking varies significantly between applications, ranging from a low of 1% to a high of 10.2% in the worst-case scenario.
  • Unexpected overhead reduction: For some weak-scaled applications (HPCG, LAMMPS), a slight decrease in overhead was observed when PEBS tracking was enabled, a phenomenon the authors note requires further investigation.
  • Access patterns are discernible: The captured data clearly reveals memory access patterns, showing distinct behaviors and hot spots within applications, which is essential for informed memory management decisions.

Key Results

The study found that PEBS-based memory access tracking on the IHK/McKernel OS incurs an average overhead of 2.3% across all tested workloads and configurations. The worst-case overhead observed was 10.2% (for GeoFEM with aggressive PEBS settings), while the best case was 1%. By adjusting PEBS parameters, the authors were able to reduce the overhead of their worst-case scenario from 10.2% to 4% while still obtaining clearly visible access patterns. The paper also shows that decreasing the PEBS reset value (e.g., from 128 to 64) significantly increases the granularity and detail of the captured access patterns.

Limitations and Future Work

Limitations: * The current PEBS implementation on the Intel Knights Landing processor primarily captures load addresses associated with L2 cache misses, not all memory accesses (e.g., stores). * The PEBS records themselves do not contain timestamps, so the timing information is derived from the interrupt handler, which can introduce some imprecision. * The observed overhead reduction in some weak-scaled applications is not fully understood and warrants further investigation.

Future Work: * The authors plan to dedicate a hardware thread to periodically harvest PEBS data instead of relying on interrupts, which could further reduce overhead. * They intend to analyze the difference between their custom PEBS driver and the Linux kernel driver to better understand limitations. * The most significant future direction is to develop runtime mechanisms that leverage the captured access patterns to actively reorganize memory pages across heterogeneous memory tiers, thereby realizing the full potential of this tracking capability for heterogeneous memory management.

Summary updated

Abstract

Operating systems have historically had to manage only a single type of memory device. The imminent availability of heterogeneous memory devices based on emerging memory technologies confronts the classic single memory model and opens a new spectrum of possibilities for memory management. Transparent data movement between different memory devices based on access patterns of applications is a desired feature to make optimal use of such devices and to hide the complexity of memory management to the end-user. However, capturing memory access patterns of an application at runtime comes at a cost, which is particularly challenging for large scale parallel applications that may be sensitive to system noise. In this work, we focus on the access pattern profiling phase prior to the actual memory relocation. We study the feasibility of using Intel's Processor Event-Based Sampling (PEBS) feature to record memory accesses by sampling at runtime and study the overhead at scale. We have implemented a custom PEBS driver in the IHK/McKernel lightweight multi-kernel operating system, one of whose advantages is minimal system interference due to the lightweight kernel's simple design compared to other OS kernels such as Linux. We present the PEBS overhead of a set of scientific applications and show the access patterns identified in noise-sensitive HPC applications. Our results show that clear access patterns can be captured with a 10% overhead in the worst-case and 1% in the best case when running on up to 128k CPU cores (2,048 Intel Xeon Phi Knights Landing nodes). We conclude that online memory access profiling using PEBS at large scale is promising for memory management in heterogeneous memory environments.
BibTeX
@inproceedings{nonell2018applicability,
  author = {Aleix Roca Nonell and Balazs Gerofi and Leonardo Bautista-Gomez and Dominique Martinet and Vicenç Beltran Querol and Yutaka Ishikawa},
  title = {{On the Applicability of PEBS based Online Memory Access Tracking for Heterogeneous Memory Management at Scale}},
  booktitle = {MCHPC@SC},
  year = {2018},
  doi = {10.1145/3286475.3286477},
}
Figure