TL;DR
This paper introduces a new, publicly available dataset of cluster workload and failure information, addressing the lack of up-to-date and comprehensive data for systems and dependability research. They demonstrate its utility by analyzing job failures, identifying potential library-related causes, and highlighting the importance of diverse data sources beyond just node-level hardware failures.
What Problem Does This Paper Address?
Managing large, high-performance computing (HPC) clusters presents several challenges. These clusters are used by a diverse range of users with varying levels of expertise, running a wide array of applications with different execution environments and resource demands. This diversity, coupled with the inherent complexity of job submission scripts and the frequent occurrence of failures (both hardware and software), makes effective cluster management difficult. Existing public datasets, like the Computer Failure Data Repository (CFDR), are often outdated and focus primarily on node-level hardware or OS failures, failing to capture the intricate interplay between workloads and their impact on performance and reliability. This paper aims to fill this gap by providing a rich, up-to-date, and anonymized dataset that can support more comprehensive analyses of cluster workloads and failures.
Methods
The authors have created a new public data repository containing workload traces from a large university community cluster. This repository includes five types of data: accounting logs (job ID, queue, timestamps, user, resources, exit status), TACC Stats (fine-grained resource usage metrics like disk, network, and process statistics), Syslog (kernel and system messages), Library lists (periodic snapshots of shared libraries accessed by jobs), and Job scripts (user-submitted job scripts). For privacy, only accounting logs and TACC Stats are made publicly available.
To demonstrate the dataset's utility, they perform a data-driven analysis of job failures. They classify job failures based on exit codes, distinguishing between successful runs, scheduler/node failures, and user-defined exit codes. They also introduce a statistical metric called "FScore" to identify libraries that might be implicated in specific types of job failures, particularly memory-related issues. This FScore is calculated based on the probability of a job using a library failing with a specific error type versus succeeding.
Datasets
The primary dataset introduced in this paper is a new, publicly available workload dataset from the "Conte" cluster at Purdue University. This dataset contains traces for 489,971 jobs and includes anonymized accounting logs and TACC Stats. The authors mention that they plan to periodically update this repository with data from other university and government lab clusters. They also reference the Computer Failure Data Repository (CFDR) as a previous effort in creating public failure repositories, noting its limitations.
Key Insights and Ideas
- The need for diverse and up-to-date data: The paper strongly emphasizes that current research and management practices are hindered by the lack of comprehensive, up-to-date datasets that capture the full spectrum of cluster activity, including workload interactions and performance metrics.
- Beyond node-level failures: Failures in large clusters are not solely due to hardware or OS issues. The interaction between diverse workloads and applications can lead to performance degradation and soft failures, which are often missed by traditional failure analysis.
- Libraries as potential failure culprits: The analysis suggests that specific libraries, or interactions between them, can be strongly correlated with certain types of job failures, such as memory exhaustion. This opens up new avenues for debugging and performance optimization.
- The value of anonymized, rich datasets: By providing a detailed and anonymized dataset, the authors enable the research community to conduct more in-depth analyses and develop better tools for cluster management and dependability.
Key Results
- 16.2% of jobs in the Conte cluster failed (exit code not equal to 0).
- The most common reasons for job failures (based on exit codes) were "Time expired (timeout)" (20.3%), "Memory exhaustion" (15.2%), and "Segmentation Fault" (9.3%).
- A significant portion (40%) of failed jobs could not be classified by their exit codes.
- The FScore analysis identified specific libraries that were highly correlated with memory-related failures, though the names of these libraries are withheld due to security reasons.
- The paper demonstrates that syslog messages can corroborate exit code-based failure classifications, with 92% of memory exhaustion failures preceded by relevant syslog messages.
Limitations and Future Work
The authors acknowledge that the exit code-based failure classification has limitations, as it doesn't always allow for fine-grained analysis. They suggest that a more thorough analysis of syslog messages might reveal more nuanced causes of failure. Future work includes periodically updating the public repository with new data and expanding it to include datasets from other clusters. They also express a desire for the research community to leverage this repository for more insightful analyses, potentially leading to improved cluster management and dependability.
Summary updated