Bookkeeping Is Not Judgement: EJBench, an Engineering Judgement Benchmark Built from a Conjugate Heat Transfer Design Campaign

James Sadler, Rotimi Alabi, Kotub UddinarXiv 2026 · arXiv

TL;DR

EJBench is a new benchmark designed to evaluate "engineering judgement"—the ability of an AI to predict physical system behavior and make design decisions based on limited data without running a simulation. Using a massive dataset of battery cold plate designs verified by physics simulations, the authors show that while frontier models like GPT-6 Astra lead the pack, they remain significantly overconfident and struggle with the nuanced intuition that human engineers develop over years of practice.

What Problem Does This Paper Address?

In software engineering, LLMs have made massive strides because training data is abundant and feedback (unit tests) is cheap and objective. Physical engineering is a different beast. Data is often proprietary and siloed, and "correctness" requires expensive, time-consuming simulations or physical experiments.

Existing engineering benchmarks typically fall into two categories: textbook/exam questions (which models can now solve with >90% accuracy) or "agentic" tasks that test if a model can operate CAD or CFD software. This paper identifies a missing middle: judgement. An experienced engineer doesn't just run simulations; they use their intuition to predict trends, spot anomalies in data, and decide which simulations are even worth running. EJBench aims to quantify this specific, non-calculative skill.

Methods

The authors built EJBench from a real-world design campaign for battery cold plates involving Conjugate Heat Transfer (CHT)—the complex interplay of heat conduction through solids and convection through fluids.

The benchmark consists of three task families:

  1. Campaign Reconstruction: Matching unlabeled result sheets to their specific operating points and identifying "misfiled" data from different designs.
  2. Evidence Sign-off: A multi-step task where the model must choose which hidden simulation results to "reveal" (within a budget) before deciding if a design meets safety limits.
  3. Operating Envelope: Ranking new operating conditions by peak cell temperature and identifying the heat-balance regime (e.g., is the pack gaining or losing heat to the surroundings?).

To ensure the benchmark measures judgement rather than simple pattern matching, the authors implemented "shortcut audits." Tasks are only included if simple linear fits or "closest-neighbor" logic would lead to the wrong answer.

Datasets

The benchmark is built on EJBench v0.1, a dataset derived from thousands of steady-state CHT simulations.

  • Training Set: 1,281 tasks for fine-tuning.
  • Validation Set: 133 tasks.
  • Test Set: 272 tasks (228 used for the official benchmark score).
  • Ground Truth: Results are generated by FluxCore (a GPU-based solver) and rigorously verified against OpenFOAM to ensure numerical accuracy.
Figure 2: FluxCore against OpenFOAM for the four task quantities: rises in kelvin, pressure drop as relative error. The grey bands mark the task tolerance.

Key Insights and Ideas

  • "Bookkeeping is not Judgement": The authors argue that being able to perform a thermodynamic calculation is just "bookkeeping." True judgement is knowing how a design will behave when you change the coolant flow rate or ambient temperature without doing the math first.
  • The Overconfidence Gap: One of the most striking findings is that models express nearly the same level of confidence in their answers regardless of whether they are right or wrong. This suggests that current LLMs lack "epistemic humility" in physical domains.
  • Physics-Verified Rewards: Because every answer is derived from a converged simulation, the benchmark can be used for Reinforcement Learning with Verifiable Rewards (RLVR). The authors demonstrated this by post-training Qwen3.5-9B, rewarding it based on how many simulation-verified criteria it met.
  • The "Pressure Drop" Trap: Models were frequently misled by pressure drop data. If a result sheet had a plausible pressure drop but an impossible temperature rise (violating the heat balance), models often failed to spot the error.

Key Results

The benchmark revealed a clear hierarchy among frontier models, though none are near "human expert" levels yet.

  • Leaderboard: GPT-6 Astra currently holds the top spot with a macro score of 0.710, followed by GPT-6.1 Sol (0.689) and Claude Opus 5.5 (0.641).
  • Open Model Performance: The base Qwen3.5-9B model struggled significantly, scoring only 0.071. However, after RL post-training using the EJBench training set, its score improved to 0.106.
  • Cost vs. Performance: The authors found that price is a poor predictor of performance. For instance, GPT-6.1 Terra outperformed several models that cost significantly more per task.
Leaderboard showing macro scores for various models
Macro score vs cost per task

Limitations and Future Work

  • Simulation vs. Reality: While verified against other solvers, the ground truth is still a simulation, not experimental data. It assumes uniform heat generation, which is a simplification of real battery behavior.
  • Scoring Rigidity: The current "all-or-nothing" scoring is quite punishing. A model that gets 7 out of 8 criteria right still receives a zero for that task.
  • Scope: v0.1 is limited to battery cold plates. The authors plan to expand the benchmark to other engineering domains and rebuild the training/validation sets as more simulations complete.
  • RL Scaling: While the RL experiment showed improvement, the gain was modest. Future work will explore more sophisticated reward functions and larger-scale training to see if open models can close the gap with frontier APIs.

Summary updated

Abstract

We introduce EJBench, a benchmark of engineering judgement: whether a model can predict how a design behaves at new conditions from a few earlier results, and decide on it. Unlike engineering benchmarks that test whether an agent can operate simulation or CAD software, EJBench is software-agnostic and asks what an engineer would conclude without running a simulation. Its tasks come from a conjugate heat transfer design campaign for battery cold plates, in three families: matching unlabelled test results to their operating points, choosing which stored runs to reveal before signing off a design, and ranking new duties by peak cell temperature. Every answer is taken from a converged simulation, not from an LLM judge. The best of ten frontier models, GPT-6 Astra, scores 0.71, and models state almost the same confidence on tasks they get wrong as on tasks they get right. We also post-train Qwen3.5-9B with LoRA and reinforcement learning, rewarded by the share of answer criteria the simulation results confirm; on the held-out test split its score rises from 0.071 to 0.106. We release the train, validation and test splits, the evaluation harness and the leaderboard.
BibTeX
@misc{sadler2026bookkeeping,
  author = {James Sadler and Rotimi Alabi and Kotub Uddin},
  title = {{Bookkeeping Is Not Judgement: EJBench, an Engineering Judgement Benchmark Built from a Conjugate Heat Transfer Design Campaign}},
  year = {2026},
  eprint = {2611.21707},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21707},
}
Figure