Reliability and Performance Trade-off Study of Heterogeneous Memories

Manish Gupta, David Roberts, Mitesh R. Meswani, Vilas Sridharan, Dean M. Tullsen, Rajesh K. GuptaMEMSYS 2016 · DOI

TL;DR

This paper investigates the critical trade-off between performance and reliability in modern heterogeneous memory systems, which combine fast, in-package memory with larger, off-package memory. The authors demonstrate that prioritizing performance alone can significantly degrade reliability over time, and propose an "aging-aware" access rate control mechanism to ensure long-term system stability.

What Problem Does This Paper Address?

Modern computing systems are increasingly adopting heterogeneous memory architectures (HMA) to overcome the limitations of traditional memory. These HMAs typically feature a combination of fast, high-bandwidth, but capacity-constrained in-package memory (like die-stacked High Bandwidth Memory - HBM) and slower, lower-bandwidth, but high-capacity off-package memory (like DDR DRAM). The primary goal of HMA research has been to maximize performance by intelligently migrating data to the fastest memory tier. However, a critical challenge arises: the in-package memory, often using less robust error correction codes (ECC) due to space and power constraints, is inherently less reliable than its off-package counterpart. This paper addresses the gap in understanding how this difference in reliability, coupled with the aging of memory components, impacts the overall system's reliability, especially in long-running, mission-critical applications. The authors highlight that systems optimized solely for peak performance might become unreliable over their operational lifespan.

Methods

The authors employ a two-pronged approach:

  • Reliability Simulation: They use FaultSim, an event-based DRAM fault simulator, to model the behavior of heterogeneous memory systems over time. FaultSim is fed with real-world DRAM failure rate data from previous field studies. The simulation models two memory tiers: an in-package memory (M1) with Single Error Correct-Double Error Detect (SEC-DED) ECC and an off-package memory (M2) with single-ChipKill ECC. The simulation spans approximately 7 years (348 weeks) and injects faults to mimic aging. The key metric for reliability is the probability of uncorrected errors.
  • Performance Simulation: They utilize Ramulator, a cycle-accurate DRAM simulator, extended to model a two-level heterogeneous memory system. They configure a 16-core system with HBM as the in-package memory and DDR3 as the off-package memory. Workloads from the SPEC benchmark suite are used, and memory traces are generated using PinPlay. The access rate to the in-package HBM memory is varied to evaluate performance impacts.

Datasets

The paper leverages real-world DRAM failure rate data from the field study by Sridharan and Liberty [31] as input for their reliability simulations. For performance evaluation, they use workloads from the SPEC benchmark suite (specifically, mcf, astar, and cactus), chosen to represent a range of memory access patterns (high, medium, and low MPKI).

Key Insights and Ideas

  • Performance vs. Reliability is a Stark Trade-off: Simply pushing more accesses to the faster, in-package memory significantly increases the probability of system failure, especially as the system ages.
  • Aging is a Critical Factor: The accumulation of permanent faults over time dramatically amplifies the negative impact of higher access rates to less reliable memory. The failure probability can increase by over 1000x in the seventh year compared to the first.
  • ECC Matters, But Isn't a Silver Bullet: While ChipKill offers much better protection than SEC-DED, its application to in-package memory is limited by physical constraints. This necessitates careful management of access patterns.
  • Access Rate as a Control Knob: The authors propose that the access rate to the faster, less reliable memory can be dynamically adjusted as a control mechanism to manage reliability without sacrificing all performance.
  • Workload Sensitivity: Different workloads exhibit varying sensitivities to access rate changes, impacting how much performance can be traded for reliability.

Key Results

  • Dramatic Reliability Degradation: Without any control, the system's failure probability can increase by a factor of 1377.77x over 7 years.
  • Performance Peaks with Balanced Access: Peak performance is achieved not by exclusively using the fastest memory, but by distributing accesses between in-package and off-package memory, leveraging the strengths of both. For some workloads, this peak occurs around a 60% access rate to the in-package memory.
  • Significant Performance Loss for Full Reliability: Restricting all accesses to the more reliable off-package memory can lead to performance degradations of 36.7% to 51.3% depending on the workload.
  • Aging-Aware Control Effectiveness: The proposed aging-aware access rate control policy, capping failure probability degradation at 300x the initial value, significantly reduces the failure probability in later years. For example, it reduces the failure probability by 26x in the final week compared to the first week.
  • Workload-Specific Aging Behavior: Workloads like cactus can sustain higher access rates for longer before hitting reliability caps, while mcf requires earlier intervention.

Limitations and Future Work

  • FIT Rate Approximation: The paper acknowledges that assuming constant FIT rates over 7 years is an approximation. While they add a margin to field data, real-world FIT rates can fluctuate.
  • Lack of Die-Stacked Field Data: The authors note that there is a lack of field study data specifically for die-stacked memory error rates. As these technologies become more prevalent, such data will be crucial for more accurate modeling.
  • Environmental Variations: The paper briefly touches upon how environmental factors (like altitude) can affect FIT rates, suggesting that future implementations of their aging-aware algorithm would need to account for such variations, potentially by logging error events via hardware mechanisms like x86's MCA registers.
  • Future Work: The authors suggest that future research should focus on filling the gaps in understanding heterogeneous memory reliability, developing more sophisticated adaptive data placement policies, and exploring hardware/software co-design for better error detection and recovery mechanisms. They also point out the significant cost associated with uncorrectable errors, including system downtime, data loss, and security vulnerabilities, underscoring the importance of their work.

Summary updated

BibTeX
@inproceedings{gupta2016reliability,
  author = {Manish Gupta and David Roberts and Mitesh R. Meswani and Vilas Sridharan and Dean M. Tullsen and Rajesh K. Gupta},
  title = {{Reliability and Performance Trade-off Study of Heterogeneous Memories}},
  booktitle = {MEMSYS},
  year = {2016},
  doi = {10.1145/2989081.2989113},
}
Figure