Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models

Toufique Ahmed, Supriyo Ghosh, Chetan Bansal, Thomas Zimmermann, Xuchao Zhang, Saravan Rajmohan

Introduction

In the modern era of hyperscale cloud computing, service availability is paramount. When a major cloud service experiences an outage or performance degradation, the financial and operational costs can escalate to millions of dollars per minute. Traditionally, resolving these incidents relies heavily on on-call engineers (OCEs) who must manually navigate complex telemetry, logs, and documentation to identify the root cause and implement a fix.

This paper, a collaborative effort between Microsoft and UC Davis, investigates the potential of Large Language Models (LLMs) to automate and accelerate this process. By focusing on two critical stages of incident management—Root Cause Analysis (RCA) and Mitigation Recommendation—the researchers provide the first large-scale evaluation of LLMs in a high-stakes production environment.

The Problem: The Manual Bottleneck in AIOps

As cloud architectures grow in complexity, traditional rule-based automation often fails to account for the myriad ways systems can break. While previous research in Artificial Intelligence for IT Operations (AIOps) has successfully addressed incident triaging and prioritization, the actual diagnosis and resolution remain significant manual bottlenecks.

The study addresses two primary research questions:

  1. Root Cause Analysis (RQ1): Can an LLM accurately identify the underlying cause of an incident based on its title and initial summary?
  2. Mitigation Recommendation (RQ2): Can the model suggest actionable steps to restore service functionality?

Methodology and Experimental Design

To answer these questions, the researchers evaluated several model families and training configurations:

Models and Fine-Tuning

The study compared OpenAI’s GPT-3 family (Curie, Codex, Davinci) and GPT-3.5 (code-davinci-002). To adapt these general-purpose models to the highly specialized domain of cloud infrastructure, the team employed Low-Rank Adaptation (LoRA) for efficient fine-tuning. These were compared against traditional encoder-decoder architectures, such as RoBERTa and CodeBERT, which served as baselines.

Training Regimes

The models were tested under three distinct settings:

  • Zero-shot: Evaluating the model's performance without any domain-specific training.
  • Fine-tuned: Training the model on historical incident data.
  • Multi-task: A single model trained to perform both root cause analysis and mitigation recommendation simultaneously.

Evaluation Framework

The researchers utilized a multi-faceted evaluation approach. Lexical metrics like BLEU-4 and ROUGE-L were used to measure text overlap, while semantic metrics such as BERTScore and NUBIA assessed the underlying meaning. Crucially, the study included a human validation phase where 25 experienced incident owners at Microsoft reviewed and rated the model's suggestions.

Dataset: Real-World Scale

The research leveraged a massive, proprietary dataset from Microsoft consisting of 44,340 incidents across 1,759 different cloud services. The data included incident titles, summaries, and the "ground truth"—the actual root causes and mitigations documented by engineers after resolution. Rigorous deduplication and filtering were applied to ensure the models learned from high-quality, non-transient data.

Key Insights

The Necessity of Fine-Tuning

While LLMs are often praised for their zero-shot capabilities, this study found that cloud incident management is too domain-specific for general knowledge to suffice. Fine-tuning resulted in a dramatic performance increase, including a 210% gain in BLEU-4 scores for mitigation tasks.

Avoiding the "Generic" Trap

A significant finding was the limitation of traditional metrics. Smaller baseline models like RoBERTa often achieved high lexical scores by producing generic responses (e.g., "The issue was self-mitigated"). While technically "correct" in some cases, these responses are useless to an engineer. In contrast, LLMs provided specific, actionable details that, while sometimes penalized by lexical metrics, were highly valued by human reviewers.

Contextual Synergy

The research demonstrated that providing the model with the root cause significantly improved the quality of its mitigation recommendations. This suggests a collaborative human-AI workflow: an engineer verifies a model-suggested root cause, which then enables the model to provide a more accurate and effective fix.

Results and Human Impact

The code-davinci-002 (GPT-3.5) model emerged as the top performer across nearly all metrics. In the human study, 70% of incident owners rated the model’s suggestions as helpful (a score of 3/5 or higher). Furthermore, the models scored exceptionally high on readability (~4/5), a critical factor for engineers working under the high pressure of a live outage. Even when the suggestions were not perfectly accurate, engineers noted they often served as a useful "guide," pointing them toward the correct investigation path.

Limitations and Future Directions

Despite the promising results, several challenges remain:

  • Signal Integration: The current models only processed text-based titles and summaries. Future iterations could benefit from incorporating live telemetry, logs, and real-time discussion threads.
  • Privacy: Handling sensitive infrastructure data requires strict protocols to prevent the model from leaking private information in its suggestions.
  • Multimodality: Cloud diagnostics often involve interpreting telemetry graphs and charts. Integrating multimodal capabilities would allow models to "see" the data that engineers rely on.

Conclusion

This research marks a significant step toward autonomous cloud incident management. By demonstrating that fine-tuned LLMs can provide readable, actionable, and helpful diagnostic support, the study paves the way for AI assistants that can significantly reduce the time and cost of maintaining global cloud services.

Summary updated

BibTeX
@inproceedings{ahmed2023recommending,
  author = {Toufique Ahmed and Supriyo Ghosh and Chetan Bansal and Thomas Zimmermann and Xuchao Zhang and Saravan Rajmohan},
  title = {{Recommending Root-Cause and Mitigation Steps for Cloud Incidents using Large Language Models}},
  booktitle = {ICSE},
  year = {2023},
  doi = {10.1109/icse48619.2023.00149},
}
Figure