Abstract
We present EPIC, an AI-driven platform designed to augment operational data analytics. EPIC employs a hierarchical multi-agent architecture where a top-level large language model provides query processing, reasoning and synthesis capabilities. These capabilities orchestrate three specialized low-level agents for information retrieval, descriptive analytics, and predictive analytics. This architecture enables EPIC to perform HPC operational analytics on multi-modal data, including text, images, and tabular formats, dynamically and iteratively. EPIC addresses the limitations of existing HPC operational analytics approaches, which rely on static methods that struggle to adapt to evolving analytics tasks and stakeholder demands. Through extensive evaluations on the Frontier HPC system, we demonstrate that EPIC effectively handles complex queries. Using descriptive analytics as a use case, fine-tuned smaller models outperform large state-of-the-art foundation models, achieving up to 26% higher accuracy. Additionally, we achieved 19x savings in LLM operational costs compared to proprietary solutions by employing a hybrid approach that combines large foundational models with fine-tuned local open-weight models.
Introduction
Modern exascale High-Performance Computing (HPC) systems, such as Oak Ridge National Laboratory’s Frontier, generate multi-terabyte streams of telemetry data, logs, and documentation daily. Traditionally, Operational Data Analytics (ODA) has relied on static dashboards and manual data engineering. This creates a significant "last-mile" bottleneck: facility managers, researchers, and administrators often lack the specialized data science expertise required to extract specific, iterative insights on demand.
Current ODA systems are often rigid, struggling to adapt to evolving questions or synthesize information across disparate formats—such as correlating a system manual's text with real-time power usage telemetry. EPIC (Generative AI Platform for Accelerating HPC Operational Data Analytics) aims to bridge this gap by providing a natural language interface capable of reasoning, querying, and predicting across multi-modal datasets.
The EPIC Architecture
The core of EPIC is a hierarchical multi-agent architecture designed to separate high-level reasoning from low-level execution. This modularity allows the system to handle complex, multi-step queries that require different types of data processing.
Specialized Agent Modules
- Query Processing Module (QP): Acting as the "manager," this module (typically powered by a high-level model like GPT-4o) interprets natural language user queries, decomposes them into logical sub-tasks, and routes them to the appropriate specialized agents.
- Information Retrieval (IR) Agent: This agent manages a RAG (Retrieval-Augmented Generation) pipeline for unstructured data like PDFs and wikis. It utilizes multimodal embeddings (CLIP) to process images and tables alongside text, ensuring that documentation is fully accessible to the system.
- Descriptive Analytics (DA) Agent: A specialized Text-to-SQL engine converts natural language into executable SQL queries for telemetry databases (e.g., DuckDB). It employs a fine-tuned Llama-3-8B model to ensure high accuracy within the specific domain of HPC telemetry schemas.
- Predictive Analytics (PA) Agent: This module translates user requests into structured parameters for traditional machine learning models (such as Neural Networks and Decision Trees) to forecast system metrics like power consumption or job duration.
Key Insights: Specialization and Efficiency
A central finding of the EPIC project is that domain-specific specialization often outweighs raw model scale. In the Descriptive Analytics module, the fine-tuned 8B parameter Llama-3 model outperformed the significantly larger GPT-4o by up to 26% in generating accurate SQL for HPC telemetry. This suggests that for technical, domain-heavy tasks, targeted fine-tuning is more effective than relying on general-purpose foundation models.
Furthermore, EPIC utilizes a hybrid model strategy. By delegating complex "routing" logic to proprietary models while using local, fine-tuned models for task execution, the system achieves a 19x reduction in operational costs. This approach also enhances data privacy by keeping sensitive telemetry processing within local infrastructure.
Experimental Results
EPIC was evaluated using several expert-vetted benchmarks, including an IR Gold Standard of 1,000 Q&A pairs and a DA dataset of 500 SQL mappings. The top-level agent achieved an F1 score of 0.77 in correctly routing queries, while subject matter experts gave the system a mean Likert score of 3.89/5 for the relevance and actionability of its synthesized responses.
As shown in the performance evaluation, the Descriptive Analytics module maintains high accuracy even as query complexity increases. The system achieved over 90% accuracy for queries requiring two joins, demonstrating the robustness of the fine-tuned local models in handling complex relational data structures common in HPC telemetry.
Conclusion and Future Work
EPIC demonstrates that a hierarchical, multi-agent approach can effectively automate the complex workflows of HPC operational analytics. While the system shows high performance in retrospective analysis, future work will focus on real-time integration with live data streams for active system monitoring. Additionally, the team is working to improve "triple-threat" query handling—tasks that require simultaneous coordination between retrieval, descriptive, and predictive modules—and enhancing the overall trustworthiness of generated code to prevent hallucinations in critical operational environments.
Summary updated