First CE Matters: On the Importance of Long Term Properties on Memory Failure Prediction

Jasmin Bogatinovski, Odej Kao, Qiao Yu, Jorge Cardoso

TL;DR

This paper argues that predicting memory failures requires looking at the entire history of correctable errors (CEs), not just recent ones. By developing new features that capture long-term CE trends, they significantly improve prediction accuracy, anticipating failures up to three hours in advance.

What Problem Does This Paper Address?

Data centers rely heavily on reliable memory. When Dynamic Random Access Memory (DRAM) fails, it can lead to data loss and system crashes. Predicting these failures allows for proactive measures like server migration or memory replacement, preventing catastrophic outages. The challenge lies in identifying early warning signs. Correctable errors (CEs), detected by ECC (Error Correcting Codes) within memory modules, are known indicators of degradation. However, previous approaches often focused on a fixed, short window of recent CEs, potentially missing crucial long-term patterns in how these errors accumulate and evolve over time, especially since memory access isn't always uniform.

Methods

The authors propose a novel approach to memory failure prediction by analyzing Correctable Errors (CEs) from hardware logs. Their core innovation lies in moving beyond fixed-size observation windows. Instead, they developed a set of features that are incrementally calculable and designed to preserve long-range temporal dependencies in CE data. This means they can capture patterns from the entire history of a DIMM's CEs without needing to store all the raw data. These features are then fed into standard machine learning classifiers, specifically Random Forest and XGBoost, to build a predictive model.

Datasets

The study was evaluated on real-world memory failure data collected from the server fleet of a large cloud provider over a six-month period (November-March 2020/21). This dataset included approximately 12,000 DIMMs, with about 3% experiencing failures. The authors mention using a heuristic to label CEs as indicative of a degraded state, based on identifying large gaps between CE bursts.

Key Insights and Ideas

  • Long-term CE history is crucial: The paper strongly suggests that the evolution of CEs over extended periods, not just recent occurrences, is key to accurate failure prediction.
  • Incremental features preserve long-range dependencies: The development of features that can be calculated incrementally is a significant contribution, allowing for the capture of historical patterns without prohibitive storage requirements.
  • Bursting nature of CEs: The authors highlight that CEs often occur in bursts, and the time between these bursts, as well as the characteristics of new bursts, are important indicators.
  • Beyond fixed windows: The limitations of fixed-size observation windows are clearly demonstrated, as they can miss critical long-term trends in CE generation.

Key Results

The proposed method, utilizing long-term CE evolution features, achieved significant improvements over traditional fixed-window approaches:

  • Average improvement: 21% in precision and 19% in recall.
  • Prediction horizon: The model can anticipate failures up to three hours in advance, which is deemed sufficient for operational interventions like server migration.
  • Outperforms operational practices: The machine learning models, leveraging the new features, outperformed manual operational practices in prediction accuracy.

Limitations and Future Work

  • Imbalanced dataset: The dataset exhibits imbalanced classification (only 3% failures), which the authors addressed using subsampling of the majority class. While effective, this is a common challenge in failure prediction.
  • Feature importance: The authors noted that features characterizing the transition between different time intervals (specifically, the interval before and during the observation window) were most important, suggesting further research into how to best leverage this transition information.
  • Future research: The paper explicitly calls for further research into how to better utilize the overall CE data to enhance memory failure prediction performance.

Summary updated

Abstract

Dynamic random access memory failures are a threat to the reliability of data centres as they lead to data loss and system crashes. Timely predictions of memory failures allow for taking preventive measures such as server migration and memory replacement. Thereby, memory failure prediction prevents failures from externalizing, and it is a vital task to improve system reliability. In this paper, we revisited the problem of memory failure prediction. We analyzed the correctable errors (CEs) from hardware logs as indicators for a degraded memory state. As memories do not always work with full occupancy, access to faulty memory parts is time distributed. Following this intuition, we observed that important properties for memory failure prediction are distributed through long time intervals. In contrast, related studies, to fit practical constraints, frequently only analyze the CEs from the last fixed-size time interval while ignoring the predating information. Motivated by the observed discrepancy, we study the impact of including the overall (long-range) CE evolution and propose novel features that are calculated incrementally to preserve long-range properties. By coupling the extracted features with machine learning methods, we learn a predictive model to anticipate upcoming failures three hours in advance while improving the average relative precision and recall for 21% and 19% accordingly. We evaluated our methodology on real-world memory failures from the server fleet of a large cloud provider, justifying its validity and practicality.
BibTeX
@inproceedings{bogatinovski2022first,
  author = {Jasmin Bogatinovski and Odej Kao and Qiao Yu and Jorge Cardoso},
  title = {{First CE Matters: On the Importance of Long Term Properties on Memory Failure Prediction}},
  booktitle = {IEEE Big Data},
  year = {2022},
  url = {https://openreview.net/forum?id=zoUJFGTY7K},
}

Loading related content...

Figure