Autonomous Multi-modal Agents for ITOps Observability

Gunjan Siddharth, Sanjana Garai, Utsav Shekhar, Prakhar Jain, Sarvam Maheshwari, Balaji Ganesan, Avirup Saha, Seema Nagar, Pratik Mishra, Felix George, Suranjana Samanta, Harshit Kumar, Mudit Verma, Prateeti Mohapatra, Ravi Jagannathan, Amit Paradkar, Sameep Mehta, Pratibha MoogiarXiv 2026 · arXiv

TL;DR

Researchers have introduced ITOps Observability Studio, a multi-agent system that uses specialized LLMs to automate complex IT operations—from generating dashboards via natural language to managing data storage costs. By using a three-tier architecture and a unified GraphQL data layer, the system successfully orchestrates workflows across fragmented tools, though it reveals that the "serving stack" (how a model is deployed) is just as critical as the model itself.

What Problem Does This Paper Address?

Modern IT Operations (ITOps) is drowning in data. To monitor system health, engineers must navigate a fragmented landscape of metrics, logs, and traces spread across dozens of products, each with its own API and naming conventions.

While Large Language Models (LLMs) are great at writing code, they often struggle with "real-world" ITOps tasks that require operating multiple professional tools in sequence. Specifically, the authors identify a gap in end-to-end automation: how do we go from a Site Reliability Engineer (SRE) asking a simple question in plain English to a fully realized, data-driven dashboard that integrates information from three different databases?

Methods

The authors propose a three-tier reference architecture to solve this:

  1. Presentation Tier: A chat-based UI that handles task decomposition and routing.
  2. Agent Coordination Tier: A "team" of specialized agents built with LangGraph. These include a Query Agent (translates text to GraphQL), a Schema Mapping Agent (connects different data sources), a Visualization Agent (creates charts), and a Data Lifecycle Management (DLM) Agent (optimizes storage costs).
  3. Tool Integration Tier: Uses the Model Context Protocol (MCP) to provide standardized access to external tools like Kubecost or databases.

A key innovation is the use of GraphQL as a unified data access layer. Instead of the agents trying to understand every raw API, they reason over a federated GraphQL schema. Furthermore, the authors treated the agents like the software they monitor, instrumenting them with OpenTelemetry to trace every LLM and tool call.

Datasets

The study evaluates the system using several benchmarks:

  • ITOps Observability Dataset: A new collection of 351 natural-language requests (e.g., "Map concert vulnerabilities to turbo entities") categorized by task type.
  • NL-to-GraphQL Benchmark: 1,164 questions over 13 schemas to test query generation.
  • Schema Mapping & Visualization: Custom sets of product schema pairs and SRE-specific queries.
  • DLM Scenarios: 30 constructed scenarios to test the agent's ability to prune data and save costs.

Key Insights and Ideas

  • The "Serving Stack" Matters: One of the most striking findings was that an agent's success isn't just about the model (e.g., Llama-3.3). Defects appeared simply because of how the "serving stack" (the gateway or chat template) formatted messages. A model might ignore tool results in one environment but work perfectly in another.
  • GraphQL as a Reasoning Buffer: By having agents interact with GraphQL schemas rather than raw data rows, the system avoids hitting LLM context limits. The agent reasons about the structure of the data, and the query engine handles the volume.
  • Observing the Observers: By applying observability to the agents themselves, the team could pinpoint exactly where a workflow failed—whether it was a bad routing decision, a hallucinated tool call, or high latency in a specific sub-agent.
  • The Hybrid Routing Trade-off: While keyword-based rules are fast and cheap, they can "shadow" the LLM. For example, if a rule sends everything starting with "Show me" to the visualization agent, it might miss a complex query that required the GraphQL agent first.

Key Results

The system was tested primarily using Llama-3.3-70B-Instruct and Granite-4-H-Small:

  • Routing: The LLM-based orchestrator achieved 94.3% accuracy in sending tasks to the right specialized agent.
  • Query Generation: On the GraphQL benchmark, the system achieved 85.7% schema-validity and a 58.4% exact match with "gold" queries.
  • Data Management: The DLM agent solved 73.3% of scenarios correctly, successfully choosing the best trace-sampling methods to balance cost and visibility.
  • Visualization: The Visualization Agent had a 0% Error Rate (meaning all generated code rendered), with high scores for "legality" (matching the user's intent).

Limitations and Future Work

The authors acknowledge several hurdles:

  • Model Retirement: During the study, the original Granite-3.x models were retired by the provider, forcing a migration. This highlights a major operational risk for agentic systems: "prompt drift" when switching models.
  • Synthetic Data: Many of the backends and requests were synthetic or small-sample, meaning the results might not fully capture the "noise" of a massive production environment.
  • Scalability: The current system uses a single orchestrator; as the number of agents grows, this could become a bottleneck.

Future Work: The team plans to explore Parameter-Efficient Fine-Tuning (PEFT) like LoRA to adapt models specifically to ITOps schemas, rather than relying solely on complex prompting.

Summary updated

Abstract

ITOps observability products must integrate data from different tools and systems. We present ITOps Observability Studio, a multi-modal multi-agent system that orchestrates observability workflows through collaborative AI agents. Its three-tier architecture features inter-agent communication, specialized LLM-powered agents for data access, schema operations and visualization, and unified tool integration. The system supports natural language to GraphQL translation, GraphQL schema generation, schema mapping, trace data lifecycle management and multi-tool workflows for reliability engineering. Its agents are instrumented with agent analytics, so that their model and tool calls can be traced. We evaluate each agent on its own benchmark, scored against gold answers where they exist and by a model judge for schema mapping and chart quality. Query generation on a public benchmark and routing work well with a current open model, and the data lifecycle management agent solves most of its scenarios. The evaluation exposed defects that only appear when prompts, parsers and serving stacks are tested together, and we report them together with their fixes.
BibTeX
@misc{siddharth2026autonomous,
  author = {Gunjan Siddharth and Sanjana Garai and Utsav Shekhar and Prakhar Jain and Sarvam Maheshwari and Balaji Ganesan and Avirup Saha and Seema Nagar and Pratik Mishra and Felix George and Suranjana Samanta and Harshit Kumar and Mudit Verma and Prateeti Mohapatra and Ravi Jagannathan and Amit Paradkar and Sameep Mehta and Pratibha Moogi},
  title = {{Autonomous Multi-modal Agents for ITOps Observability}},
  year = {2026},
  eprint = {2611.19982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19982},
}
Figure