TL;DR
LLMs are increasingly acting as autonomous scientists, but can we trust their work? This paper introduces DFT-Agent Bench, a rigorous benchmark for density-functional-theory agents that includes a "provenance audit" to ensure results come from real simulations rather than hallucinations. The study finds that while sophisticated tool-use "scaffolds" boost performance in complex tasks like electronic structure calculations, the most challenging physics problems remain unsolved by current agents.
What Problem Does This Paper Address?
As Large Language Models (LLMs) evolve into autonomous agents, they are being tasked with end-to-end scientific workflows—preparing inputs, running specialized codes like Quantum ESPRESSO (QE), and parsing complex results. However, two major questions remain:
- Engineering vs. Intelligence: How much does "agent engineering" (adding tools, verifiers, and domain knowledge) actually help compared to just prompting a state-of-the-art model?
- Provenance: In science, the process is as important as the result. Can we trust that an agent's high score is the product of a valid, completed simulation, or did it just "hallucinate" a correct-looking answer?
Existing benchmarks often ignore the execution contract, focusing only on the final numerical output. This paper addresses this by introducing a benchmark that evaluates both the outcome and the integrity of the execution process.
Methods
The authors developed a comparison framework using two different agent architectures:
- v1 (Baseline): A "near-bare" ReAct loop with four generic tools (write file, run shell, read output, submit result).
- v5.1 (Flagship): An engineered scaffold featuring structured QE tools, input linting, a domain-knowledge layer, an SCF (self-consistent field) convergence sub-agent, and a verifier gate.
Crucially, they implemented a four-flag execution audit (provenance gate) that checks:
qe_invoked: Did the binary actually run?job_done: Did the full methodology grid complete with QE's completion markers?qe_wallclock_ok: Was the execution time physically plausible?no_network_access: Was the environment isolated?
If a run fails any of these, its score is set to zero, regardless of how "correct" the final report looks.
Datasets
The study introduces DFT-Agent Bench, consisting of 200+ Quantum ESPRESSO tasks categorized by:
- Material Tiers: Ranging from M1 (simple elemental/binary solids) to M2 (complex oxides) and M3 (magnetic transition-metal systems requiring spin-polarized calculations).
- Task Types: T1 (variable-cell relaxation), T2 (band structure and density of states), T3 (equation-of-state fitting), and T4 (elastic tensor calculation).
Ground truth was established via converged QE runs, cross-validated against established public references like the Materials Project.
Key Insights and Ideas
- The Provenance Gate is Essential: The audit caught cases where agents produced correct-looking answers despite incomplete simulations (e.g., only 3 out of 7 required points for a curve). Without this gate, roughly 2% of runs would have bypassed the execution requirements.
- Scaffold Saturation: For "easier" tasks like simple relaxation (T1) and equation-of-state (T3), both the baseline and engineered agents scored near ~0.9. In these regimes, the model's inherent capability is the bottleneck, and extra engineering provides little numerical headroom.
- Targeted Gains in Complexity: Scaffolding shines in specific, complex workflows. For example, the v5.1 scaffold significantly improved performance in band structure calculations (T2) by enforcing proper metal band-path protocols that the bare baseline often missed.
- Model-Dependent Engineering: The impact of scaffolding varies by model. For the difficult elastic tensor task, DeepSeek-v4-pro saw a massive jump (0.18 to 0.40) with the scaffold, whereas Claude-Opus-4-8's baseline was already higher and didn't benefit as much.
Key Results
The engineered v5.1 scaffold provided clear benefits in specific regimes:
- Electronic Structure (Bands+DOS): Improved scores by +0.08 (DeepSeek) and +0.11 (Claude).
- Magnetic Systems (M3 vc-relax): Both models saw a gain of +0.17.
- Elastic Tensors: A bimodal result where DeepSeek improved significantly (+0.22), while Claude remained flat (see Figure 1).
As shown in Table 1, the "gated" scores (which account for the provenance audit) are often lower than "raw" scores, particularly for M1 relaxation tasks, highlighting the audit's role in catching execution failures.
Limitations and Future Work
- Unsolved Frontiers: Even with heavy scaffolding, the hardest tasks—like elastic tensors and magnetic band structures—remain largely unsolved, with scores topping out around 0.6-0.7.
- Component Attribution: The study compares "packages" (v1 vs v5.1). Future work is needed to perform fine-grained ablations to see exactly which component (e.g., the verifier vs. the domain knowledge) contributes most to the gains.
- Data Contamination: Since the benchmark uses public materials data, the authors acknowledge they cannot rule out that some task content appeared in the models' pretraining data.
- Stochasticity: Single agent runs are noisy; the authors suggest that at least three repeats are necessary for stable cell-level conclusions.
Summary updated