AskALCF: A Self-Improving Agentic System for Intelligent HPC User Support

Jingyan Jiang, Akhilesh Bondapalli, Krishna Rauniyar, Murat Keceli, Yasaman Ghadar, Haritha Siddabathuni Som, Venkatram Vishwanath, Michael E. Papka, Rong Ge, Huihuo ZhengarXiv 2026 · arXiv

TL;DR

AskALCF is a framework that transforms AI support logs into a roadmap for system improvement. By using a diagnostic pipeline to pinpoint exactly why an answer failed—distinguishing between missing documentation, retrieval errors, or model hallucinations—it enables a "governed self-evolution" loop where human experts oversee continuous, data-driven upgrades to the support system.

What Problem Does This Paper Address?

High-Performance Computing (HPC) facilities like the Argonne Leadership Computing Facility (ALCF) are incredibly complex environments where software, hardware, and procedures change almost daily. While Retrieval-Augmented Generation (RAG) systems (AI chatbots that "read" manuals) have made support more accessible, they face two major hurdles:

  1. Opaque Failures: When a chatbot gives a bad answer, it’s often unclear why. Is the information missing from the documentation? Did the search engine fail to find it? Or did the AI model simply misunderstand the context?
  2. Static Deployment: Most systems remain static after they are launched. They might get documentation updates, but the underlying logic, prompts, and retrieval strategies rarely evolve based on real-world user interactions.

The authors address these gaps by creating a system that treats every user interaction as evidence for targeted engineering repairs.

Methods

The AskALCF architecture is built on four "Pillars" that connect live Q&A with long-term knowledge management:

  • Pillar 1 (Q&A): The front-end RAG service.
  • Pillar 3 (Log Analysis): The "brain" of the improvement loop. It uses a two-scope sufficiency test to evaluate failures. It asks: R (Was the retrieved info enough?) and C (Does the answer exist anywhere in the entire corpus?).
  • The Diagnostic Core: An ordered rule tree assigns failures to one of five owners: Knowledge (missing docs), Retrieval (search failed), Prompt (bad instructions), Model (reasoning error), or Tools (technical glitch).
  • Agentic Audit: An "Agent Judge" independently probes the documentation corpus to verify the rule tree's decisions, acting as an automated auditor rather than a replacement.
  • Harness Training: Instead of retraining the AI model's weights, the authors "train" the harness—the collection of rules, prompts, and judge instructions. This is done through a controlled process of proposing changes and testing them against frozen expert labels.
Fig. 1. AskALCF connects two support channels to a governed improvement loop. A user query enters Pillar 1 (1), whose trace is analyzed by Pillar 3 (2). Confirmed knowledge gaps go to Pillar 2 (3), while other failures go to their component owners (3b). A ticket enters Pillar 4 (0), which drafts a reply for staff review (4) and sends reusable fixes to Pillar 2 (5). Pillar 2 opens a pull request (6); after human approval, the merge trigger re-indexes the corpus (7). That final trigger has not yet fired.

Datasets

The study utilized a massive corpus of real-world data from the ALCF:

  • Production Logs: 2,174 scored assistant turns across 1,229 user sessions.
  • Expert Reference: A manually labeled set of 150 turns (split into 98 training and 52 evaluation) used to benchmark the diagnostic pipeline.
  • Support Tickets: 2,211 ticket threads used to analyze how human-led support overlaps with AI-led support.
  • Repair Sets: Specific subsets (288 for prompts, 68 for knowledge) used for controlled offline experiments.

Key Insights and Ideas

  • The $R \times C$ Pivot: The most actionable insight is distinguishing between a "Knowledge" failure and a "Retrieval" failure. If the info exists but wasn't found ($R=False, C=True$), you fix the search engine. If it doesn't exist at all ($C=False$), you call a technical writer.
  • Harness vs. Weights: In production environments, you often can't (or shouldn't) retrain a massive LLM. AskALCF shows that you can achieve significant gains by optimizing the "harness"—the logic that surrounds the model.
  • Governed Self-Evolution: The system proposes repairs, but humans retain the "Merge Gate." This ensures that the AI doesn't autonomously change facility policy or documentation without expert oversight.
  • The "Unknown" is the Enemy: A major goal of the system was reducing "unattributable" failures. If you don't know why it failed, you can't fix it.
Fig. 2. The diagnostic workflow and its expert-review loop. Layers B and A produce retrieval and answer evidence, respectively. The ordered rule tree (3) assigns the initial route, and the agent judge (4) audits that decision. A human expert reviews the evidence and both rationales (5); those labels support harness changes to the rule tree and judge skills (6). The lower panels show the review interface for a turn on which the two routes disagree.

Key Results

The authors demonstrated that their diagnostic approach leads to measurable system improvements:

  • Better Attribution: The pipeline reduced unattributable ("unknown") failures from 41.4% to 14.2% by improving evidence coverage.
  • Expert Agreement: Harness training increased the system's agreement with human experts from 108/150 to 123/150 turns.
  • Identifying the Bottleneck: The system revealed that 76% of missing facts were actually already in the documentation; they just weren't being retrieved correctly. This shifted the team's priority from writing new docs to improving search algorithms.
  • Successful Repairs: In controlled experiments, targeted prompt changes significantly reduced "over-clarification" (the AI asking unnecessary follow-up questions), while injecting missing documentation pages raised fact utilization from 0.070 to 0.202.
Failure mode distribution before and after training

Limitations and Future Work

While AskALCF provides a robust framework, it is not yet a "fully closed" autonomous loop.

  • Human Bottleneck: Knowledge changes still require manual approval and have not yet been auto-merged into the live production index.
  • Data Specificity: All data came from a single facility (ALCF). While the design lessons are transferable, the specific rules and prompts might need tuning for other environments.
  • Rare Failures: Some failure modes (like tool execution errors) were too rare in the dataset to provide statistically significant accuracy metrics.

Future work will focus on automating the retrieval-side repairs and evaluating the final stage of the loop: the autonomous re-indexing of approved knowledge changes.

Summary updated

Abstract

AI-based question-answering systems are increasingly used for user support, including at high-performance computing (HPC) facilities. However, existing systems lack a systematic, deployment-oriented approach for evaluating final-answer quality and attributing failures to specific components. They also remain largely static after deployment, apart from periodic knowledge-base updates. We present AskALCF, an agentic log-analysis and improvement pipeline for HPC user support. AskALCF evaluates production interactions at a fine-grained level, identifies whether failures arise from missing knowledge, retrieval, prompting, model behavior, or tool execution, and routes each failure to an actionable repair. Its diagnostic core combines answer-quality assessment with a two-scope sufficiency test that distinguishes information missing from the documentation corpus from information present but not retrieved. We further introduce a controlled harness-training process that uses expert references, null edits, and explicit ship, revert, or decline decisions to improve the diagnostic pipeline reliably. Across 2,174 support turns, AskALCF reduced unattributable failures from 41.4% to 14.2%, while harness training improved expert agreement from 108/150 to 123/150 turns. These results demonstrate a governed form of self-evolution in which production evidence drives targeted system improvements. By turning production interactions into fine-grained diagnoses and actionable repairs, AskALCF provides a systematic foundation for continuously improving deployed AI support systems beyond knowledge-base updates alone.
BibTeX
@misc{jiang2026askalcf,
  author = {Jingyan Jiang and Akhilesh Bondapalli and Krishna Rauniyar and Murat Keceli and Yasaman Ghadar and Haritha Siddabathuni Som and Venkatram Vishwanath and Michael E. Papka and Rong Ge and Huihuo Zheng},
  title = {{AskALCF: A Self-Improving Agentic System for Intelligent HPC User Support}},
  year = {2026},
  eprint = {2611.15700},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15700},
}
Figure