CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development

Zhiyu Zhou, Tianyou Liu, Xinping Yang, Yiyao Liu, Jiayi Dong, Yihan Lin, Haokai Ye, Yifan Lu, Zeyu Dong, Yansong Li, Shengqi ZhangarXiv 2026 · arXiv

TL;DR

CAE-Bench is a new benchmark designed to test whether AI agents can develop reusable Computer-Aided Engineering (CAE) solvers that generalize to unseen geometries and boundary conditions. The results are a wake-up call: even top-tier models like GPT-6 Astra, which ace fixed problems, struggle significantly when faced with held-out cases, often failing to produce numerically accurate results for complex fluid dynamics.

What Problem Does This Paper Address?

In the world of computational science, AI agents have become surprisingly good at writing code to solve a specific Partial Differential Equation (PDE) with a fixed set of parameters. However, an industrial-grade CAE solver (like those used to design car aerodynamics or heat sinks) needs to be reusable. It must handle varying geometries, material properties, and boundary conditions without needing a human to rewrite the code for every new simulation.

Existing benchmarks often provide the agent with all the inputs during the development phase. This allows agents to "cheat" by essentially hardcoding the answer for those specific inputs. CAE-Bench addresses this by strictly separating the development phase (where agents see a few "public" cases) from the evaluation phase (where the solver is tested on "held-out" cases it has never seen).

Methods

The authors propose a rigorous two-stage workflow for evaluating AI agents:

  1. Development Phase: The agent receives a task description (e.g., "solve 2D steady incompressible flow") and a few public cases with random output values to establish the required data format. The agent then develops a single solver (a solve.py script).
  2. Evaluation Phase: The submitted solver is frozen. It is then run against multiple held-out cases featuring different physical inputs (e.g., different viscosities or domain shapes).
Figure 1: The workflow of CAE-Bench

To score these solvers, the benchmark uses a Weighted Normalized Root-Mean-Square Error (NRMSE). If a solver crashes, exceeds resource limits, or produces invalid data types, it receives a score of zero. If it runs, its accuracy is mapped to a score between 0 and 1, where an error ($E$) $\le 0.05$ is a perfect score and $E \ge 0.10$ is a zero.

Datasets

The researchers introduced the RC7 release of CAE-Bench, which includes:

  • 80 Computational Tasks: Divided into prescribed PDEs (13), fluid mechanics (43), and fixed-problem tasks (24).
  • 630 Total Cases: 70 public cases for development and 560 held-out cases for testing generalization.
  • Diverse Physics: Tasks cover everything from simple heat conduction to complex viscoelastic flow and turbulence.
  • Reference Data: Results are pulled from analytical solutions and established simulation software like OpenFOAM, Elmer, and SPARTA.
Table 1: The relationship between fixed task aspects and case-specific inputs

Key Insights and Ideas

  • The Generalization Gap: There is a massive performance drop between public and held-out cases. A model might achieve a perfect score on the cases it "practiced" on but fail completely when the geometry changes slightly.
  • Fluid Mechanics is the "Final Boss": While agents perform relatively well on basic PDEs (scoring up to 87.27), their performance plummets on fluid mechanics tasks (peaking at only 40.25).
  • Reasoning Effort Diminishing Returns: Increasing the "reasoning effort" setting on models like GPT-6 Luna doesn't always help. In fact, at "max" effort, models sometimes fail to submit a solver at all because they run out of time or get stuck in loops.
  • Numerical Accuracy is the Main Hurdle: Most zero-scores aren't caused by code crashes (execution failure) but by "scoring rejection"—meaning the code ran successfully, but the physics were wrong.
Figure 2: Input variations for periodic advection and generalized-Newtonian flow tasks.

Key Results

The authors evaluated 12 state-of-the-art models, including the GPT-6 series, Gemini 4 Argon, and DeepSeek V4.1.

  • Top Performers: GPT-6.1 Sol and GPT-6 Astra led the pack, but even they only achieved average scores of ~50% on tasks with varying physical inputs.
  • Fixed vs. Variable: On fixed-problem tasks (where no generalization is required), several models achieved 100%. However, when those same models were tested on held-out cases for the same tasks, their scores dropped to around 67%.
  • Failure Distribution: For OpenAI models, the majority of failures were due to poor numerical accuracy (scoring rejection). In contrast, models like DeepSeek and Qwen had much higher rates of submission or execution failure.
Figure 3: Case outcomes across the 80 computational tasks in RC7.

Limitations and Future Work

The authors acknowledge several limitations:

  • Domain Scope: The benchmark currently focuses heavily on PDEs and fluid mechanics; other CAE areas like structural mechanics or electromagnetics are less represented.
  • Data Contamination: Since reference results come from well-known datasets (like PDEBench), there is a risk that these solutions were part of the models' training data.
  • Validation: Not all reference results have been independently validated for absolute numerical precision.

Future Work: The team hopes to expand the task library and encourages the community to use their provided SDK to develop agents that don't just solve problems, but build robust, reusable engineering tools.

Summary updated

Abstract

We developed CAE-Bench, a benchmark to evaluate AI agents on Computer-Aided Engineering (CAE) solver development. A CAE solver is more than a PDE solver; it must be able to generalize across various inputs, such as different boundary conditions, geometries, and material properties. CAE-Bench evaluates the generalizability of AI-developed CAE solvers by providing the same task, such as a heat transfer problem, multiple cases, with each case specifying an input. The AI agent develops a solver for a fixed task using only a few public cases, and the resulting solver is then tested on held-out cases, i.e., cases it has never seen before. We evaluate 12 models across a range of reasoning settings on 80 computational tasks. Our experiments show that even if a solver developed by an AI agent performs well on the public cases, it may even get 0 score on held-out cases. For tasks such as fluid dynamics problems, even the solver developed by GPT-6 Astra performs poorly on held-out cases. A data format and an SDK for task registration are provided, so the researcher can use the benchmark on their own infrastructure. We hope that CAE-Bench will help the community to develop AI agents that can develop reusable CAE solvers that generalize to various inputs.
BibTeX
@misc{zhou2026cae,
  author = {Zhiyu Zhou and Tianyou Liu and Xinping Yang and Yiyao Liu and Jiayi Dong and Yihan Lin and Haokai Ye and Yifan Lu and Zeyu Dong and Yansong Li and Shengqi Zhang},
  title = {{CAE-Bench: Benchmarking AI Agents for Reusable CAE Solver Development}},
  year = {2026},
  eprint = {2611.21273},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21273},
}
Figure