Orbit: An Evidence-Grounded Benchmark for Evaluating LLMs on IT Operations Reasoning

Sathwik Narkedimilli, Rakshit Verma, Rahul Soni, Ankit Kushwaha, Balaji Ganesan, Kaustabha Ray, Sunil Saumya, Pratibha MoogiarXiv 2026 · arXiv

TL;DR

The authors introduce ORBIT, a new evidence-grounded benchmark designed to evaluate how well Large Language Models (LLMs) can reason through complex IT Operations (ITOps) and Site Reliability Engineering (SRE) tasks. Their findings reveal a significant "competence gap": while current models excel at multiple-choice questions through fact recall, they struggle significantly with open-ended troubleshooting and root-cause analysis.

What Problem Does This Paper Address?

In modern cloud environments, a single misconfiguration in a template can ripple through a Kubernetes cluster, manifesting as a vague latency spike in an unrelated service. Solving these "incidents" requires more than just memorizing facts; it requires causal reasoning—linking deployment history, metrics, and documentation into a coherent chain of evidence.

Existing ITOps benchmarks often fall short because:

  1. Lack of Traceability: They provide answers but don't link them to specific source passages or reasoning steps. If a model fails, you don't know why.
  2. Keyword Collapse: Many questions can be "gamed" by simple keyword lookups rather than actual reasoning.
  3. Data Privacy: Realistic production logs are often confidential, making it hard to build public, high-quality benchmarks.

ORBIT tackles these by providing a dataset where 96% of items include a recorded source, allowing researchers to trace failures back to missing facts or broken logic.

Methods

The authors developed a human-in-the-loop pipeline to build ORBIT:

  • Knowledge Base Assembly: They ingested documentation from major Application Performance Monitoring (APM) tools (Instana, Dynatrace, Datadog), cloud provider docs, and practitioner forums like Stack Overflow.
  • Retrieval-Grounded Generation: Using models like Gemini Flash, they generated questions directly from retrieved evidence packages. This ensures that every answer is grounded in specific documentation.
  • Expert Validation: Three authors with SRE experience reviewed every item for correctness and operational validity.
  • Independent Judging: To avoid the "self-enhancement" bias where models favor their own writing style, the authors used an independent LLM judge (Claude Sonnet 5.5) to score open-ended responses against a strict rubric.

Datasets

The paper introduces the ORBIT benchmark, which consists of 3,019 evaluation items across three formats:

  • Multiple-Choice (MCQ): 2,513 items testing recognition and recall.
  • Open-Ended (OE): 298 items requiring free-text explanations.
  • Troubleshooting (TS): 208 items rebuilt from real-world practitioner threads (Stack Overflow, Server Fault) to simulate complex root-cause analysis.

ORBIT is compared against other benchmarks like OpsEval and ITBench in the following table:

Table 1: Comparison of ORBIT with other ITOps benchmarks on grounding, reasoning, and traceability.

Key Insights and Ideas

  • The MCQ Mirage: Multiple-choice accuracy is a deceptive metric. Models that score ~88% on MCQs often drop to ~15-25% on troubleshooting tasks. MCQs often test "recognition," while real SRE work requires "generation" and "synthesis."
  • Surface Cues are Dangerous: The authors found that simply choosing the option with the most word overlap with the question yields a 57.9% accuracy rate on MCQs. This suggests many benchmarks are inadvertently testing pattern matching rather than IT expertise.
  • Diagnosis is the Weakest Link: Models struggle most with "abductive root cause" reasoning—inferring the most likely cause from a set of symptoms.
  • Don't Let Models Grade Themselves: When models acted as their own judges, they accepted up to 75% of answers to defective items. Using an independent, strict judge is critical for a realistic assessment.

Key Results

The authors evaluated five mid-sized models, including Llama 3.3 70B, Mistral Medium, and Granite 4.0 H.

Table 2: Zero-shot accuracy (%) of each model on the evaluation set.

As shown in Figure 2, while Mistral Medium and Llama 3.3 70B lead in overall and MCQ accuracy, their performance craters in the Troubleshooting (TS) category. Interestingly, Granite 4.0 H Small performed best on open-ended questions despite being the weakest on MCQs, highlighting how model rankings can shift dramatically depending on the task format.

The difficulty labels in ORBIT also proved highly predictive. "Hard" items (those requiring multi-step reasoning) saw a ~30-point drop in accuracy compared to standard retrieval items across all models:

Table 3: Multiple-choice accuracy (%) on standard and hard items, with Wilson confidence intervals.

Furthermore, diagnostic tasks—the core of SRE work—are significantly more challenging than general IT knowledge questions:

Table 4: Accuracy on diagnosis items vs. other items.

Limitations and Future Work

  • Closed-Book Evaluation: This study evaluated models based on their internal knowledge. In practice, SRE assistants would use Retrieval-Augmented Generation (RAG). Future work will test ORBIT in an "open-book" setting.
  • Data Imbalance: The current corpus is dominated by "lookup" style questions (62.6%), reflecting the nature of technical documentation.
  • Public vs. Private Sources: Only 25% of the items cite public web pages; the rest cite internal repositories, which limits external reproducibility for those specific items.
  • Frontier Models: The study did not include "frontier" models (like GPT-4o or Claude 3.5 Sonnet) as subjects, leaving the current "accuracy ceiling" of the benchmark unknown.

Summary updated

Abstract

LLMs now support IT operations (ITOps), but benchmarks rarely show why a model fails. We present ORBIT, a benchmark of ITOps questions in three formats, namely multiple-choice, open-ended, and troubleshooting, with each question accompanied by a reference answer and a recorded source. An audit found template defects in about a tenth of the generated items, so we drop them and rebuild the troubleshooting format from real practitioner threads with accepted answers. Under one independent judge, five LLMs answer most multiple-choice items correctly, between a third and just over half of the open-ended items, and a minority of the troubleshooting items, even under a more lenient root-cause rubric. When models judged their own answers, they accepted many answers to defective items, so we use one independent judge for every model. Word overlap with the question finds the correct option for more than half of the multiple-choice items. ORBIT is a diagnostic aid for selecting and supervising assistants rather than a deployment test.
BibTeX
@misc{narkedimilli2026orbit,
  author = {Sathwik Narkedimilli and Rakshit Verma and Rahul Soni and Ankit Kushwaha and Balaji Ganesan and Kaustabha Ray and Sunil Saumya and Pratibha Moogi},
  title = {{Orbit: An Evidence-Grounded Benchmark for Evaluating LLMs on IT Operations Reasoning}},
  year = {2026},
  eprint = {2611.15718},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15718},
}
Figure