OpsEval: A Comprehensive Task-Oriented AIOps Benchmark for Large Language Models

Yuhe Liu, Changhua Pei, Longlong Xu, Bohan Chen, Mingze Sun, Zhirui Zhang, Yongqian Sun, Shenglin Zhang, Kun Wang, Haiming Zhang, Jianhui Li, Gaogang Xie, Xidao Wen, Xiaohui Nie, Dan Pei

Large Language Models (LLMs) have demonstrated remarkable capabilities in general-purpose tasks, yet their utility in specialized "vertical" domains like IT Operations (Ops) remains under-scrutinized. The domain of AIOps is characterized by intricate technical jargon, sensitive proprietary data, and a critical requirement for precision. A model providing incorrect technical advice in a production environment can lead to catastrophic system failures.

To address this gap, the research paper "OpsEval: A Comprehensive Benchmark Suite for Evaluating Large Language Models’ Capability in IT Operations Domain" introduces a rigorous framework designed to assess LLMs within the high-stakes context of server management, network configuration, and fault analysis.

The Challenge of Benchmarking AIOps

General-purpose benchmarks such as MMLU or C-Eval are insufficient for the Ops domain for several reasons:

  1. Data Privacy: Operational data, including logs and incident tickets, is often proprietary, limiting the exposure of LLMs to real-world scenarios during pre-training.
  2. Domain Complexity: "Ops" encompasses diverse sub-domains like 5G, cloud computing, and database management, each requiring specific mental models and technical depth.
  3. The "Sounding Right" Trap: Traditional NLP metrics like BLEU and ROUGE measure lexical overlap. In Ops, a model might produce a grammatically correct and technically "sounding" response that contains a logically flawed or dangerous command.

The OpsEval Methodology

The researchers developed a comprehensive pipeline to transform raw operational data into a standardized, multi-faceted benchmark. This process involves three primary stages: data collection from diverse sources, rigorous preprocessing, and a multi-dimensional evaluation.

The OpsEval pipeline: from data collection and preprocessing to comprehensive evaluation.

Taxonomy and Ability Levels

The benchmark categorizes questions into eight distinct tasks, ranging from Automation Scripts to Network Configuration. Furthermore, it evaluates models across three ability levels:

  • Knowledge Recall: Assessing the model's ability to retrieve factual information.
  • Analytical Thinking: Testing the model's capacity to reason through complex operational scenarios.
  • Practical Application: Evaluating the model's ability to generate actionable solutions, such as scripts or configuration commands.

Evaluation and Prompting Techniques

To ensure a robust assessment, the researchers utilized various prompting strategies, including Chain-of-Thought (CoT), Self-Consistency (SC), and In-Context Learning (ICL).

Details of the evaluation phase, including question types, supported languages, and prompting techniques.

The FAE-Score: A Novel Metric

A significant contribution of the paper is the FAE-Score, a metric designed to overcome the limitations of traditional NLP evaluations in technical domains. It assesses Question-Answering (QA) performance across three dimensions:

  • Fluency: Evaluated by an LLM judge (Qwen2-72B) to ensure grammatical coherence.
  • Accuracy: Verified through keyword matching to ensure the inclusion of critical technical points.
  • Evidence: Measuring alignment with authoritative reference documents to ground answers in factual reality.

The FAE-Score demonstrated a 0.9185 correlation coefficient with human expert judgments, significantly outperforming BLEU and ROUGE, which sometimes showed negative correlations.

Dataset Composition

The OpsEval dataset comprises 9,070 bilingual questions (English and Chinese), including 7,334 multiple-choice and 1,736 open-ended QA pairs. Data was sourced from 11 collaborating companies, professional certification exams, and authoritative IT textbooks. To prevent data leakage and maintain the integrity of the leaderboard, only 20% of the dataset is publicly available, with the remainder reserved for undisclosed evaluation.

Key Findings

The evaluation of 24 leading LLMs yielded several critical insights:

  • The Reasoning Gap: While models excel at Knowledge Recall, they struggle with Analytical Thinking, particularly in generating logic-heavy Automation Scripts.
  • Performance Leaders: GPT-4 and Claude-3-Opus emerged as top performers, though open-source models like Qwen2-72B-Instruct and GLM-4 showed competitive results.
  • Quantization Thresholds: For local deployment, 4-bit quantization (INT4) maintains performance, but 3-bit (INT3) leads to a significant drop in accuracy for specialized Ops tasks.

Conclusion

OpsEval provides the first comprehensive, task-oriented benchmark for LLMs in the IT Operations domain. By introducing the FAE-Score and a diverse, bilingual dataset, the researchers have established a more reliable standard for measuring AI readiness in production environments. Future work will focus on expanding the benchmark to evaluate Ops Agents capable of tool use and real-time command execution.

Summary updated

BibTeX
@inproceedings{liu2025opseval,
  author = {Yuhe Liu and Changhua Pei and Longlong Xu and Bohan Chen and Mingze Sun and Zhirui Zhang and Yongqian Sun and Shenglin Zhang and Kun Wang and Haiming Zhang and Jianhui Li and Gaogang Xie and Xidao Wen and Xiaohui Nie and Dan Pei},
  title = {{OpsEval: A Comprehensive Task-Oriented AIOps Benchmark for Large Language Models}},
  booktitle = {SIGSOFT FSE Companion},
  year = {2025},
  doi = {10.1145/3696630.3728572},
}
Figure