TL;DR
AskALCF is a framework that transforms AI support logs into a roadmap for system improvement. By using a diagnostic pipeline to pinpoint exactly why an answer failed—distinguishing between missing documentation, retrieval errors, or model hallucinations—it enables a "governed self-evolution" loop where human experts oversee continuous, data-driven upgrades to the support system.
What Problem Does This Paper Address?
High-Performance Computing (HPC) facilities like the Argonne Leadership Computing Facility (ALCF) are incredibly complex environments where software, hardware, and procedures change almost daily. While Retrieval-Augmented Generation (RAG) systems (AI chatbots that "read" manuals) have made support more accessible, they face two major hurdles:
- Opaque Failures: When a chatbot gives a bad answer, it’s often unclear why. Is the information missing from the documentation? Did the search engine fail to find it? Or did the AI model simply misunderstand the context?
- Static Deployment: Most systems remain static after they are launched. They might get documentation updates, but the underlying logic, prompts, and retrieval strategies rarely evolve based on real-world user interactions.
The authors address these gaps by creating a system that treats every user interaction as evidence for targeted engineering repairs.
Methods
The AskALCF architecture is built on four "Pillars" that connect live Q&A with long-term knowledge management:
- Pillar 1 (Q&A): The front-end RAG service.
- Pillar 3 (Log Analysis): The "brain" of the improvement loop. It uses a two-scope sufficiency test to evaluate failures. It asks: R (Was the retrieved info enough?) and C (Does the answer exist anywhere in the entire corpus?).
- The Diagnostic Core: An ordered rule tree assigns failures to one of five owners: Knowledge (missing docs), Retrieval (search failed), Prompt (bad instructions), Model (reasoning error), or Tools (technical glitch).
- Agentic Audit: An "Agent Judge" independently probes the documentation corpus to verify the rule tree's decisions, acting as an automated auditor rather than a replacement.
- Harness Training: Instead of retraining the AI model's weights, the authors "train" the harness—the collection of rules, prompts, and judge instructions. This is done through a controlled process of proposing changes and testing them against frozen expert labels.
Datasets
The study utilized a massive corpus of real-world data from the ALCF:
- Production Logs: 2,174 scored assistant turns across 1,229 user sessions.
- Expert Reference: A manually labeled set of 150 turns (split into 98 training and 52 evaluation) used to benchmark the diagnostic pipeline.
- Support Tickets: 2,211 ticket threads used to analyze how human-led support overlaps with AI-led support.
- Repair Sets: Specific subsets (288 for prompts, 68 for knowledge) used for controlled offline experiments.
Key Insights and Ideas
- The $R \times C$ Pivot: The most actionable insight is distinguishing between a "Knowledge" failure and a "Retrieval" failure. If the info exists but wasn't found ($R=False, C=True$), you fix the search engine. If it doesn't exist at all ($C=False$), you call a technical writer.
- Harness vs. Weights: In production environments, you often can't (or shouldn't) retrain a massive LLM. AskALCF shows that you can achieve significant gains by optimizing the "harness"—the logic that surrounds the model.
- Governed Self-Evolution: The system proposes repairs, but humans retain the "Merge Gate." This ensures that the AI doesn't autonomously change facility policy or documentation without expert oversight.
- The "Unknown" is the Enemy: A major goal of the system was reducing "unattributable" failures. If you don't know why it failed, you can't fix it.
Key Results
The authors demonstrated that their diagnostic approach leads to measurable system improvements:
- Better Attribution: The pipeline reduced unattributable ("unknown") failures from 41.4% to 14.2% by improving evidence coverage.
- Expert Agreement: Harness training increased the system's agreement with human experts from 108/150 to 123/150 turns.
- Identifying the Bottleneck: The system revealed that 76% of missing facts were actually already in the documentation; they just weren't being retrieved correctly. This shifted the team's priority from writing new docs to improving search algorithms.
- Successful Repairs: In controlled experiments, targeted prompt changes significantly reduced "over-clarification" (the AI asking unnecessary follow-up questions), while injecting missing documentation pages raised fact utilization from 0.070 to 0.202.
Limitations and Future Work
While AskALCF provides a robust framework, it is not yet a "fully closed" autonomous loop.
- Human Bottleneck: Knowledge changes still require manual approval and have not yet been auto-merged into the live production index.
- Data Specificity: All data came from a single facility (ALCF). While the design lessons are transferable, the specific rules and prompts might need tuning for other environments.
- Rare Failures: Some failure modes (like tool execution errors) were too rare in the dataset to provide statistically significant accuracy metrics.
Future work will focus on automating the retrieval-side repairs and evaluating the final stage of the loop: the autonomous re-indexing of approved knowledge changes.
Summary updated