TL;DR
EJBench is a new benchmark designed to evaluate "engineering judgement"—the ability of an AI to predict physical system behavior and make design decisions based on limited data without running a simulation. Using a massive dataset of battery cold plate designs verified by physics simulations, the authors show that while frontier models like GPT-6 Astra lead the pack, they remain significantly overconfident and struggle with the nuanced intuition that human engineers develop over years of practice.
What Problem Does This Paper Address?
In software engineering, LLMs have made massive strides because training data is abundant and feedback (unit tests) is cheap and objective. Physical engineering is a different beast. Data is often proprietary and siloed, and "correctness" requires expensive, time-consuming simulations or physical experiments.
Existing engineering benchmarks typically fall into two categories: textbook/exam questions (which models can now solve with >90% accuracy) or "agentic" tasks that test if a model can operate CAD or CFD software. This paper identifies a missing middle: judgement. An experienced engineer doesn't just run simulations; they use their intuition to predict trends, spot anomalies in data, and decide which simulations are even worth running. EJBench aims to quantify this specific, non-calculative skill.
Methods
The authors built EJBench from a real-world design campaign for battery cold plates involving Conjugate Heat Transfer (CHT)—the complex interplay of heat conduction through solids and convection through fluids.
The benchmark consists of three task families:
- Campaign Reconstruction: Matching unlabeled result sheets to their specific operating points and identifying "misfiled" data from different designs.
- Evidence Sign-off: A multi-step task where the model must choose which hidden simulation results to "reveal" (within a budget) before deciding if a design meets safety limits.
- Operating Envelope: Ranking new operating conditions by peak cell temperature and identifying the heat-balance regime (e.g., is the pack gaining or losing heat to the surroundings?).
To ensure the benchmark measures judgement rather than simple pattern matching, the authors implemented "shortcut audits." Tasks are only included if simple linear fits or "closest-neighbor" logic would lead to the wrong answer.
Datasets
The benchmark is built on EJBench v0.1, a dataset derived from thousands of steady-state CHT simulations.
- Training Set: 1,281 tasks for fine-tuning.
- Validation Set: 133 tasks.
- Test Set: 272 tasks (228 used for the official benchmark score).
- Ground Truth: Results are generated by FluxCore (a GPU-based solver) and rigorously verified against OpenFOAM to ensure numerical accuracy.
Key Insights and Ideas
- "Bookkeeping is not Judgement": The authors argue that being able to perform a thermodynamic calculation is just "bookkeeping." True judgement is knowing how a design will behave when you change the coolant flow rate or ambient temperature without doing the math first.
- The Overconfidence Gap: One of the most striking findings is that models express nearly the same level of confidence in their answers regardless of whether they are right or wrong. This suggests that current LLMs lack "epistemic humility" in physical domains.
- Physics-Verified Rewards: Because every answer is derived from a converged simulation, the benchmark can be used for Reinforcement Learning with Verifiable Rewards (RLVR). The authors demonstrated this by post-training Qwen3.5-9B, rewarding it based on how many simulation-verified criteria it met.
- The "Pressure Drop" Trap: Models were frequently misled by pressure drop data. If a result sheet had a plausible pressure drop but an impossible temperature rise (violating the heat balance), models often failed to spot the error.
Key Results
The benchmark revealed a clear hierarchy among frontier models, though none are near "human expert" levels yet.
- Leaderboard: GPT-6 Astra currently holds the top spot with a macro score of 0.710, followed by GPT-6.1 Sol (0.689) and Claude Opus 5.5 (0.641).
- Open Model Performance: The base Qwen3.5-9B model struggled significantly, scoring only 0.071. However, after RL post-training using the EJBench training set, its score improved to 0.106.
- Cost vs. Performance: The authors found that price is a poor predictor of performance. For instance, GPT-6.1 Terra outperformed several models that cost significantly more per task.
Limitations and Future Work
- Simulation vs. Reality: While verified against other solvers, the ground truth is still a simulation, not experimental data. It assumes uniform heat generation, which is a simplification of real battery behavior.
- Scoring Rigidity: The current "all-or-nothing" scoring is quite punishing. A model that gets 7 out of 8 criteria right still receives a zero for that task.
- Scope: v0.1 is limited to battery cold plates. The authors plan to expand the benchmark to other engineering domains and rebuild the training/validation sets as more simulations complete.
- RL Scaling: While the RL experiment showed improvement, the gain was modest. Future work will explore more sophisticated reward functions and larger-scale training to see if open models can close the gap with frontier APIs.
Summary updated