DFT-Agent Bench: Scaffold Effects and Execution Provenance in Autonomous DFT

Shijie Liu, Jin Yang, Weilin Wan, Jingtao Han, Jia Li, Tianqi DengarXiv 2026 · arXiv

TL;DR

LLMs are increasingly acting as autonomous scientists, but can we trust their work? This paper introduces DFT-Agent Bench, a rigorous benchmark for density-functional-theory agents that includes a "provenance audit" to ensure results come from real simulations rather than hallucinations. The study finds that while sophisticated tool-use "scaffolds" boost performance in complex tasks like electronic structure calculations, the most challenging physics problems remain unsolved by current agents.

What Problem Does This Paper Address?

As Large Language Models (LLMs) evolve into autonomous agents, they are being tasked with end-to-end scientific workflows—preparing inputs, running specialized codes like Quantum ESPRESSO (QE), and parsing complex results. However, two major questions remain:

  1. Engineering vs. Intelligence: How much does "agent engineering" (adding tools, verifiers, and domain knowledge) actually help compared to just prompting a state-of-the-art model?
  2. Provenance: In science, the process is as important as the result. Can we trust that an agent's high score is the product of a valid, completed simulation, or did it just "hallucinate" a correct-looking answer?

Existing benchmarks often ignore the execution contract, focusing only on the final numerical output. This paper addresses this by introducing a benchmark that evaluates both the outcome and the integrity of the execution process.

Methods

The authors developed a comparison framework using two different agent architectures:

  • v1 (Baseline): A "near-bare" ReAct loop with four generic tools (write file, run shell, read output, submit result).
  • v5.1 (Flagship): An engineered scaffold featuring structured QE tools, input linting, a domain-knowledge layer, an SCF (self-consistent field) convergence sub-agent, and a verifier gate.

Crucially, they implemented a four-flag execution audit (provenance gate) that checks:

  • qe_invoked: Did the binary actually run?
  • job_done: Did the full methodology grid complete with QE's completion markers?
  • qe_wallclock_ok: Was the execution time physically plausible?
  • no_network_access: Was the environment isolated?

If a run fails any of these, its score is set to zero, regardless of how "correct" the final report looks.

Datasets

The study introduces DFT-Agent Bench, consisting of 200+ Quantum ESPRESSO tasks categorized by:

  • Material Tiers: Ranging from M1 (simple elemental/binary solids) to M2 (complex oxides) and M3 (magnetic transition-metal systems requiring spin-polarized calculations).
  • Task Types: T1 (variable-cell relaxation), T2 (band structure and density of states), T3 (equation-of-state fitting), and T4 (elastic tensor calculation).

Ground truth was established via converged QE runs, cross-validated against established public references like the Materials Project.

Key Insights and Ideas

  • The Provenance Gate is Essential: The audit caught cases where agents produced correct-looking answers despite incomplete simulations (e.g., only 3 out of 7 required points for a curve). Without this gate, roughly 2% of runs would have bypassed the execution requirements.
  • Scaffold Saturation: For "easier" tasks like simple relaxation (T1) and equation-of-state (T3), both the baseline and engineered agents scored near ~0.9. In these regimes, the model's inherent capability is the bottleneck, and extra engineering provides little numerical headroom.
  • Targeted Gains in Complexity: Scaffolding shines in specific, complex workflows. For example, the v5.1 scaffold significantly improved performance in band structure calculations (T2) by enforcing proper metal band-path protocols that the bare baseline often missed.
  • Model-Dependent Engineering: The impact of scaffolding varies by model. For the difficult elastic tensor task, DeepSeek-v4-pro saw a massive jump (0.18 to 0.40) with the scaffold, whereas Claude-Opus-4-8's baseline was already higher and didn't benefit as much.

Key Results

The engineered v5.1 scaffold provided clear benefits in specific regimes:

  • Electronic Structure (Bands+DOS): Improved scores by +0.08 (DeepSeek) and +0.11 (Claude).
  • Magnetic Systems (M3 vc-relax): Both models saw a gain of +0.17.
  • Elastic Tensors: A bimodal result where DeepSeek improved significantly (+0.22), while Claude remained flat (see Figure 1).
Figure 1: Model-matched v1 vs. v5.1 with 95% task-clustered bootstrap CIs (M1+M2 common tasks; no cross-model pooling). Three cells separate: bands+DOS on both models and elastic on deepseek-v4-pro only; vc-relax and EOS are flat, and the opus bare baseline already reaches 0.62 on elastic (Finding 3).

As shown in Table 1, the "gated" scores (which account for the provenance audit) are often lower than "raw" scores, particularly for M1 relaxation tasks, highlighting the audit's role in catching execution failures.

Table 1: Historical/descriptive pooled summary of per-cell raw and provenance-gated scores, v1 vs. v5.1, pooled over both models: best score per retained historical model–task record. The n values are retained historical pool entries, not unique released task IDs: the six M1+M2 rows sum to 552 entries per arm because repeats and task versions are retained, whereas the released set has 245 unique IDs. Raw and gated nearly coincide except M1 vc-relax, where flagship contract misses are voided (0.97 vs 0.91; Finding 5). The pooled elastic row averages a bimodal split (0.18→0.40 deepseek vs 0.62→0.59 opus; Fig. 1); M3 rows are per-model in Fig. 2. M2 multi-step cells (n=10) omitted.

Limitations and Future Work

  • Unsolved Frontiers: Even with heavy scaffolding, the hardest tasks—like elastic tensors and magnetic band structures—remain largely unsolved, with scores topping out around 0.6-0.7.
  • Component Attribution: The study compares "packages" (v1 vs v5.1). Future work is needed to perform fine-grained ablations to see exactly which component (e.g., the verifier vs. the domain knowledge) contributes most to the gains.
  • Data Contamination: Since the benchmark uses public materials data, the authors acknowledge they cannot rule out that some task content appeared in the models' pretraining data.
  • Stochasticity: Single agent runs are noisy; the authors suggest that at least three repeats are necessary for stable cell-level conclusions.

Summary updated

Abstract

Large language models are increasingly deployed as autonomous agents for first-principles materials simulation. We report empirical findings from DFT-Agent Bench, a benchmark of 200+ Quantum ESPRESSO tasks spanning relaxation, electronic structure, equation-of-state, and elastic-tensor pipelines over multiple material tiers. Each task carries a QE-derived ground truth with public-reference cross-checks where available, and a provenance audit checks whether the required QE execution contract was met. Under one execution harness, we compare a near-bare ReAct baseline with a bundled v5.1 scaffold on two models. The complete scaffold improves bands+DOS by +0.08/+0.11 and d-electron vc-relax by +0.17 on both models, while high-scoring relaxation and equation-of-state cells leave little room for improvement. The elastic result is model-dependent (0.18 to 0.40 for deepseek-v4-pro and 0.62 to 0.59 for claude-opus-4-8). A fixed 100-task analysis with retrieval disabled supports the aggregate pattern: the fully frozen DeepSeek comparison gains +0.17, while the Opus comparison gains +0.10 against a historical pool baseline (p < 0.001). The provenance gate catches incomplete or unmarked executions that can receive high rubric scores; it is an execution-contract check, not a test for intentional fraud. These results characterize the evaluated DFT workflows. The anonymous review artifact contains the frozen diagnostic slice, prompts, scorer, and solver revisions: https://anonymous.4open.science/r/231e31re91287h/README.md.
BibTeX
@misc{liu2026dft,
  author = {Shijie Liu and Jin Yang and Weilin Wan and Jingtao Han and Jia LI and Tianqi Deng},
  title = {{DFT-Agent Bench: Scaffold Effects and Execution Provenance in Autonomous DFT}},
  year = {2026},
  eprint = {2611.10629},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10629},
}
Figure