TL;DR
CAE-Bench is a new benchmark designed to test whether AI agents can develop reusable Computer-Aided Engineering (CAE) solvers that generalize to unseen geometries and boundary conditions. The results are a wake-up call: even top-tier models like GPT-6 Astra, which ace fixed problems, struggle significantly when faced with held-out cases, often failing to produce numerically accurate results for complex fluid dynamics.
What Problem Does This Paper Address?
In the world of computational science, AI agents have become surprisingly good at writing code to solve a specific Partial Differential Equation (PDE) with a fixed set of parameters. However, an industrial-grade CAE solver (like those used to design car aerodynamics or heat sinks) needs to be reusable. It must handle varying geometries, material properties, and boundary conditions without needing a human to rewrite the code for every new simulation.
Existing benchmarks often provide the agent with all the inputs during the development phase. This allows agents to "cheat" by essentially hardcoding the answer for those specific inputs. CAE-Bench addresses this by strictly separating the development phase (where agents see a few "public" cases) from the evaluation phase (where the solver is tested on "held-out" cases it has never seen).
Methods
The authors propose a rigorous two-stage workflow for evaluating AI agents:
- Development Phase: The agent receives a task description (e.g., "solve 2D steady incompressible flow") and a few public cases with random output values to establish the required data format. The agent then develops a single solver (a
solve.pyscript). - Evaluation Phase: The submitted solver is frozen. It is then run against multiple held-out cases featuring different physical inputs (e.g., different viscosities or domain shapes).
To score these solvers, the benchmark uses a Weighted Normalized Root-Mean-Square Error (NRMSE). If a solver crashes, exceeds resource limits, or produces invalid data types, it receives a score of zero. If it runs, its accuracy is mapped to a score between 0 and 1, where an error ($E$) $\le 0.05$ is a perfect score and $E \ge 0.10$ is a zero.
Datasets
The researchers introduced the RC7 release of CAE-Bench, which includes:
- 80 Computational Tasks: Divided into prescribed PDEs (13), fluid mechanics (43), and fixed-problem tasks (24).
- 630 Total Cases: 70 public cases for development and 560 held-out cases for testing generalization.
- Diverse Physics: Tasks cover everything from simple heat conduction to complex viscoelastic flow and turbulence.
- Reference Data: Results are pulled from analytical solutions and established simulation software like OpenFOAM, Elmer, and SPARTA.
Key Insights and Ideas
- The Generalization Gap: There is a massive performance drop between public and held-out cases. A model might achieve a perfect score on the cases it "practiced" on but fail completely when the geometry changes slightly.
- Fluid Mechanics is the "Final Boss": While agents perform relatively well on basic PDEs (scoring up to 87.27), their performance plummets on fluid mechanics tasks (peaking at only 40.25).
- Reasoning Effort Diminishing Returns: Increasing the "reasoning effort" setting on models like GPT-6 Luna doesn't always help. In fact, at "max" effort, models sometimes fail to submit a solver at all because they run out of time or get stuck in loops.
- Numerical Accuracy is the Main Hurdle: Most zero-scores aren't caused by code crashes (execution failure) but by "scoring rejection"—meaning the code ran successfully, but the physics were wrong.
Key Results
The authors evaluated 12 state-of-the-art models, including the GPT-6 series, Gemini 4 Argon, and DeepSeek V4.1.
- Top Performers: GPT-6.1 Sol and GPT-6 Astra led the pack, but even they only achieved average scores of ~50% on tasks with varying physical inputs.
- Fixed vs. Variable: On fixed-problem tasks (where no generalization is required), several models achieved 100%. However, when those same models were tested on held-out cases for the same tasks, their scores dropped to around 67%.
- Failure Distribution: For OpenAI models, the majority of failures were due to poor numerical accuracy (scoring rejection). In contrast, models like DeepSeek and Qwen had much higher rates of submission or execution failure.
Limitations and Future Work
The authors acknowledge several limitations:
- Domain Scope: The benchmark currently focuses heavily on PDEs and fluid mechanics; other CAE areas like structural mechanics or electromagnetics are less represented.
- Data Contamination: Since reference results come from well-known datasets (like PDEBench), there is a risk that these solutions were part of the models' training data.
- Validation: Not all reference results have been independently validated for absolute numerical precision.
Future Work: The team hopes to expand the task library and encourages the community to use their provided SDK to develop agents that don't just solve problems, but build robust, reusable engineering tools.
Summary updated