TL;DR
Researchers have introduced ITOps Observability Studio, a multi-agent system that uses specialized LLMs to automate complex IT operations—from generating dashboards via natural language to managing data storage costs. By using a three-tier architecture and a unified GraphQL data layer, the system successfully orchestrates workflows across fragmented tools, though it reveals that the "serving stack" (how a model is deployed) is just as critical as the model itself.
What Problem Does This Paper Address?
Modern IT Operations (ITOps) is drowning in data. To monitor system health, engineers must navigate a fragmented landscape of metrics, logs, and traces spread across dozens of products, each with its own API and naming conventions.
While Large Language Models (LLMs) are great at writing code, they often struggle with "real-world" ITOps tasks that require operating multiple professional tools in sequence. Specifically, the authors identify a gap in end-to-end automation: how do we go from a Site Reliability Engineer (SRE) asking a simple question in plain English to a fully realized, data-driven dashboard that integrates information from three different databases?
Methods
The authors propose a three-tier reference architecture to solve this:
- Presentation Tier: A chat-based UI that handles task decomposition and routing.
- Agent Coordination Tier: A "team" of specialized agents built with LangGraph. These include a Query Agent (translates text to GraphQL), a Schema Mapping Agent (connects different data sources), a Visualization Agent (creates charts), and a Data Lifecycle Management (DLM) Agent (optimizes storage costs).
- Tool Integration Tier: Uses the Model Context Protocol (MCP) to provide standardized access to external tools like Kubecost or databases.
A key innovation is the use of GraphQL as a unified data access layer. Instead of the agents trying to understand every raw API, they reason over a federated GraphQL schema. Furthermore, the authors treated the agents like the software they monitor, instrumenting them with OpenTelemetry to trace every LLM and tool call.
Datasets
The study evaluates the system using several benchmarks:
- ITOps Observability Dataset: A new collection of 351 natural-language requests (e.g., "Map concert vulnerabilities to turbo entities") categorized by task type.
- NL-to-GraphQL Benchmark: 1,164 questions over 13 schemas to test query generation.
- Schema Mapping & Visualization: Custom sets of product schema pairs and SRE-specific queries.
- DLM Scenarios: 30 constructed scenarios to test the agent's ability to prune data and save costs.
Key Insights and Ideas
- The "Serving Stack" Matters: One of the most striking findings was that an agent's success isn't just about the model (e.g., Llama-3.3). Defects appeared simply because of how the "serving stack" (the gateway or chat template) formatted messages. A model might ignore tool results in one environment but work perfectly in another.
- GraphQL as a Reasoning Buffer: By having agents interact with GraphQL schemas rather than raw data rows, the system avoids hitting LLM context limits. The agent reasons about the structure of the data, and the query engine handles the volume.
- Observing the Observers: By applying observability to the agents themselves, the team could pinpoint exactly where a workflow failed—whether it was a bad routing decision, a hallucinated tool call, or high latency in a specific sub-agent.
- The Hybrid Routing Trade-off: While keyword-based rules are fast and cheap, they can "shadow" the LLM. For example, if a rule sends everything starting with "Show me" to the visualization agent, it might miss a complex query that required the GraphQL agent first.
Key Results
The system was tested primarily using Llama-3.3-70B-Instruct and Granite-4-H-Small:
- Routing: The LLM-based orchestrator achieved 94.3% accuracy in sending tasks to the right specialized agent.
- Query Generation: On the GraphQL benchmark, the system achieved 85.7% schema-validity and a 58.4% exact match with "gold" queries.
- Data Management: The DLM agent solved 73.3% of scenarios correctly, successfully choosing the best trace-sampling methods to balance cost and visibility.
- Visualization: The Visualization Agent had a 0% Error Rate (meaning all generated code rendered), with high scores for "legality" (matching the user's intent).
Limitations and Future Work
The authors acknowledge several hurdles:
- Model Retirement: During the study, the original Granite-3.x models were retired by the provider, forcing a migration. This highlights a major operational risk for agentic systems: "prompt drift" when switching models.
- Synthetic Data: Many of the backends and requests were synthetic or small-sample, meaning the results might not fully capture the "noise" of a massive production environment.
- Scalability: The current system uses a single orchestrator; as the number of agents grows, this could become a bottleneck.
Future Work: The team plans to explore Parameter-Efficient Fine-Tuning (PEFT) like LoRA to adapt models specifically to ITOps schemas, rather than relying solely on complex prompting.
Summary updated