Sibylla: To Retry or Not To Retry on Deep Learning Job Failure

Taeyoon Kim, Suyeon Jeong, Jongseop Lee, Soobee Lee, Myeongjae JeonUSENIX ATC 2022 · OpenReview

TL;DR

Deep learning training jobs often fail and are blindly retried by schedulers, wasting valuable GPU resources. This paper introduces Sibylla, a machine learning system that predicts whether a failed deep learning job is likely to succeed upon retry. By intelligently deciding when to retry, Sibylla significantly improves cluster utilization and reduces job completion times.

What Problem Does This Paper Address?

Deep learning (DL) training demands substantial computational resources, particularly GPUs, which are often shared in large clusters. When a DL job fails, current cluster schedulers typically have a simple retry mechanism. This can lead to wasted GPU cycles if the failure is deterministic (e.g., a bug in the code or data) and will likely recur. Conversely, blindly aborting jobs that experience non-deterministic failures (e.g., temporary network issues) can reduce overall training success rates and require users to restart from scratch, also wasting previous computation. The problem is to distinguish between these failure types to make informed retry decisions, thereby optimizing resource usage and job completion times.

Methods

Sibylla employs a recurrent neural network (RNN)-based machine learning model to classify job failures. The core idea is to train on the standard output (stdout) and standard error (stderr) logs generated by DL training jobs.

Here's a breakdown of the key steps:

  • Data Preprocessing: Log files are first processed to extract relevant information. Sibylla takes the last 20 lines from a log file, focusing on lines containing keywords like "failure" or "error," and includes a few preceding lines to capture context.
  • Parsing and Vectorization: Raw log messages are unstructured. Sibylla uses the Drain tool to parse these logs into structured templates, removing irrelevant words. These templates are then converted into semantic vectors using TF-IDF scoring and the FastText algorithm to capture word importance and semantic meaning.
  • Model Training: Bi-directional Long Short-Term Memory (LSTM) and Attention-based Gated Recurrent Unit (GRU) models are used for training. Sibylla employs a semi-supervised learning approach, starting with partially labeled data and automatically labeling new data through an "auto-labeling" mechanism.
  • Auto-Labeling and Ensemble: To handle the continuous stream of new log data without manual labeling, Sibylla uses an ensemble method. It trains multiple RNN models independently and aggregates their predictions through majority voting to make a robust classification decision (deterministic vs. non-deterministic failure). This ensemble approach helps mitigate the impact of individual model mispredictions.
  • Integration: Sibylla is designed to be integrated into cluster managers. When a job fails, the cluster manager sends the relevant logs to Sibylla, which returns a prediction. The manager can then use this prediction to decide whether to retry the job.

Datasets

The paper uses two main sources of data:

  • Microsoft Philly Trace: This trace dataset provides information about job scheduling attempts, allocated GPUs, execution times, and final job status. It's used for evaluating the effectiveness of Sibylla in a simulated environment.
  • Custom Failure Log Dataset: To train Sibylla's classifier, the authors constructed a dataset by:
  • Collecting 97 failure log files from a company operating a "NoRetry" system.
  • Gathering an additional 159 failure messages from Stack Overflow, covering various failure categories.
  • Applying data augmentation techniques (WordNet and Word2Vec) to enlarge the dataset and balance DT and NDT samples, resulting in a total of 4468 log files.

Key Insights and Ideas

  • The "Retry or Not Retry" Dilemma: The core insight is that not all failures are equal. Deterministic failures will repeat, making retries wasteful, while non-deterministic failures can be resolved by retrying.
  • Log Data as a Rich Source: Stdout and stderr logs, often overlooked for their unstructured nature, contain crucial information for predicting failure types.
  • The Power of Semi-Supervised Learning and Auto-Labeling: Manually labeling the vast amount of log data generated by DL clusters is infeasible. Sibylla's auto-labeling mechanism, combined with an ensemble of RNN models, provides a practical way to continuously improve the classifier without human intervention.
  • Ensemble for Robustness: Using an ensemble of models (LSTM and GRU in this case) significantly improves prediction accuracy and robustness compared to individual models, especially when dealing with noisy or ambiguous log data.
  • Predictive Retry as a Key to Efficiency: The paper demonstrates that intelligently applying retries based on failure prediction can lead to substantial improvements in cluster utilization and job completion times.

Key Results

  • Classifier Performance: Sibylla achieved high accuracy in classifying deterministic (DT) and non-deterministic (NDT) failures. In the final round of incremental training, Sibylla achieved:
  • DT Precision: 97.36%

    • DT Recall: 98.66%
    • NDT Precision: 96.92%
    • NDT Recall: 97.32% Sibylla's performance closely approached that of an "oracle" (perfectly labeled data).
  • Job Completion Time (JCT) Reduction: Through trace-driven simulations, Sibylla reduced the average JCT by:

  • Best Case: 6.5% to 15.4%

    • Worst Case: 3.6% to 10.5% These reductions were observed across different job scheduling policies (SJF, DLAS, GITTINS).
  • Negligible Impact on Success Rate: Sibylla's high recall for NDT failures (98.66%) meant that the job success rate was only negligibly impacted (0.06% drop from 75.04%).

Limitations and Future Work

  • Misprediction of New Failure Types: The paper acknowledges that if a completely new type of failure with unseen semantics appears, Sibylla's accuracy can drop significantly (e.g., DT precision dropping to 40% in a specific scenario). The authors suggest that incorporating human experts for labeling new data with low prediction confidence might be necessary in such cases.
  • Improving Repetitive Mispredictions: For deterministic failures, if a job consistently produces similar error messages across retries, mispredictions might still occur. Future work could involve revising the classifier to incorporate feedback on predictive re-execution from the cluster scheduler to address this.
  • Extending to New Failure Types: While the current system is trained on known failure types, future work could explore methods to better handle entirely novel failure patterns.

Summary updated

BibTeX
@inproceedings{kim2022sibylla,
  author = {Taeyoon Kim and Suyeon Jeong and Jongseop Lee and Soobee Lee and Myeongjae Jeon},
  title = {{Sibylla: To Retry or Not To Retry on Deep Learning Job Failure}},
  booktitle = {USENIX ATC},
  year = {2022},
  url = {https://openreview.net/forum?id=Gxwnp2LF4z},
}
Figure