AIChilles: Automatically Uncovering Hidden Weaknesses in AI-Evolved Systems

Yajie Zhou, Ao Li, Ashwin Silla, Zaoxing Liu, Vyas SekararXiv 2026 · arXiv

TL;DR

AI-driven system optimization (ADSO) frameworks can automatically rewrite code to achieve impressive benchmark gains, but they often introduce severe hidden weaknesses like system crashes, resource blowups, and silent performance regressions on unseen workloads. To expose these vulnerabilities, AIChilles introduces an automated, agentic framework that uses the original human-written code as a differential testing oracle combined with quality-diversity search. Across 30 AI-evolved programs, AIChilles uncovered 49 distinct hidden weaknesses, demonstrating that integrating automated adversarial auditing directly into the AI development lifecycle is essential for building robust systems.

What Problem Does This Paper Address?

Recent AI-driven system optimization frameworks (such as AdaEvolve and Engram) use Large Language Models (LLMs) to iteratively rewrite core system algorithms—like database transaction schedulers or load balancers—boasting 12–60% performance gains over human designs.

However, these frameworks evaluate candidate code using a fixed set of benchmark workloads. This creates a massive blind spot: AI agents excel at "reward hacking" and overfitting to the specific evaluation metrics, often replacing clean, lightweight human heuristics with highly complex code. When these AI-evolved programs encounter real-world, unseen workloads, they can suffer from catastrophic scalability regressions, memory leaks, or outright crashes. Traditional software testing techniques (like manual fuzzing or symbolic execution) fail to solve this because they require significant human effort to set up custom harnesses for every single generated program variant.

Methods

AIChilles automates the discovery of these "Achilles' heels" using a multi-stage agentic workflow:

  1. Semantic-Aware Workload Space Inference: Instead of relying purely on LLM prompting to guess valid inputs, AIChilles uses Python AST parsing to deterministically extract workload variables from the system's evaluator. It then tasks an LLM agent to infer the semantic constraints between these variables, generating a valid workload grammar.
  2. Type-Specific Sub-Agents: Rather than using a single monolithic agent that might get stuck repeatedly finding the same easy crash, AIChilles splits the search among four specialized sub-agents, each targeting a specific regression type: Correctness, Scalability-Time, Scalability-Memory, and Optimality.
  3. Divergence-Guided Optimization via MAP-Elites: AIChilles treats the original human-written program ($P$) as a differential oracle. It runs both $P$ and the AI-evolved program ($P'$) on generated workloads, prioritizing inputs where $P'$ performs significantly worse than $P$.
  4. Execution Trajectory Diversity: To ensure the framework uncovers diverse root causes rather than minor variations of the same bug, it tracks the line-by-line code execution frequencies of $P'$. It uses these frequency vectors as a behavioral diversity signal within a MAP-Elites evolutionary algorithm to map out the failure space.
Figure 7: Divergence-guided weakness search.

Datasets

The authors evaluated AIChilles across 5 representative system applications sourced from recent literature:

  • TXN-Sched: A database transaction order optimizer.
  • EPLB: An expert parallelism load balancer for Mixture-of-Experts (MoE) inference.
  • Cloudcast: A cost-aware multi-cloud data transfer scheduler.
  • LLM-SQL: A prompt-ordering optimizer designed to maximize prefix-cache hit rates.
  • Prism: A cache-aware model placement policy for multi-LLM serving.

These applications were optimized using 3 different ADSO frameworks (OpenEvolve, AdaEvolve, Engram) and 2 frontier LLMs (GPT-5 and Claude-Opus-4.6), yielding 30 distinct AI-evolved programs for evaluation.

Key Insights and Ideas

  • The Complexity Trap: LLMs frequently trade simple $O(N)$ or $O(M+G)$ greedy human heuristics for complex global optimization logic (e.g., deep local searches, caching intermediate states, or dense cost matrices). While this wins on small benchmark workloads, it causes algorithmic blowups (e.g., scaling to $O(N^2)$ or $O(MG^2)$) on larger inputs.
  • Silent Quality Regressions: In applications like Prism, AI-evolved programs rarely crashed; instead, they introduced silent optimality regressions where they made significantly worse placement decisions under specific GPU cluster configurations than the simple human baseline.
  • Behavioral Diversity Matters: Traditional fuzzing relies on input parameter distance to maintain diversity. AIChilles proves that tracking code execution frequency is a far superior proxy for behavioral diversity, allowing the framework to catch performance bottlenecks caused by loops being heavily executed under specific load conditions.
Execution time and balance score regressions under changing workloads

Key Results

AIChilles successfully uncovered 49 distinct hidden weaknesses across the 30 program configurations.

  • Failure Breakdown: The most frequent issues were execution-time blowups (25 instances), followed by memory regressions (11), correctness/crashes (7), and optimality drops (6).
  • Efficiency: Compared to traditional baselines (Random Fuzzing, Mutational Fuzzing, Property-Based Testing, and a Naive LLM Agent), AIChilles found significantly more diverse weaknesses within a fixed 6-hour budget, often converging on its final set of distinct bugs within just 20–30 minutes.
  • Mitigation Trade-off: When the authors integrated AIChilles directly into the AdaEvolve optimization loop as a continuous integration (CI) gatekeeper, it successfully filtered out all flawed code candidates. However, this robustness came at a cost: the final optimized programs reverted closer to the human baseline, shrinking the original "claimed" benchmark improvements from 19% down to 0% on Prism, and from 49% down to 3.9% on TXN-Sched.

Limitations and Future Work

While AIChilles successfully automates the discovery and grouping of adversarial workloads, its root-cause analysis stage still relies on an LLM agent to interpret the anomalous execution profiles. The authors note that these LLM-generated explanations can sometimes be inconsistent, occasionally describing a nearby symptom rather than pinpointing the exact underlying algorithmic flaw. They suggest that future iterations could combine the LLM agent with deterministic debugging techniques like dynamic program slicing or delta debugging to ground the root-cause reports in concrete execution evidence.

Summary updated

Abstract

The computer systems community has recently seen growing interest in AI-driven system evolution, where AI agents iteratively rewrite systems. Frameworks such as AdaEvolve and Engram report 12-60% score improvements over human-designed algorithms. While these results are promising, there are practical concerns if these AI-evolved programs can perform worse on unseen workloads and exhibit scalability regressions. Given the speed and scale of AI-generated code, we need automated mechanisms to uncover such identify hidden weaknesses in AI-evolved systems programs. To this end, we develop AIChilles that takes as input a baseline program $P$ and an AI-evolved program $P'$, AIChilles searches for valid workloads where $P'$ regresses relative to $P$ in correctness, runtime, memory usage, or output quality. To tackle the diversity in system applications, weakness types and potential bugs, AIChilles combines deterministic workload-parameter extraction, agent-based constraint inference, differential oracles, and code-frequency coverage to discover diverse failures. Across five system applications and 30 AI-evolved programs, AIChilles finds 49 distinct hidden weaknesses. We also show that explicitly including AIChilles in the AI-driven development lifecycle can mitigate several of these weaknesses.
BibTeX
@misc{zhou2026aichilles,
  author = {Yajie Zhou and Ao Li and Ashwin Silla and Zaoxing Liu and Vyas Sekar},
  title = {{AIChilles: Automatically Uncovering Hidden Weaknesses in AI-Evolved Systems}},
  year = {2026},
  eprint = {2606.15834},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.15834},
}
Figure