FrontierPhysics: Evaluating Agents for End-to-End Physics Research

Bingran You, Wenjun Ke, Ria Rosenauer, Bo Liu, Wenbo Chen, Min Chen, Shuqi Xu, Yue Ma, Jiechao Feng, Bo Chen, Xiangkai Sun, Jiarui Liu, Jinen Guo, Zeyu Zhou, Xu Huang, Zhuohan Li, Haoyu Li, Heguang Lin, Lijie Ding, Yadong Zeng, Kangqi Zhang, Xinge Liu, Hanwen Liu, Zhenghong Li, Alexander von Recum, Yuchen Zhu, Felix Schwarzfischer, Yizhou Zhao, Jicheng Wang, Binxu Li, Qian Yu, Chan Li, Cheng Dai, Di Wang, Haoyu Zhang, Jiankai Sun, Kaituo Feng, Mian Qin, Ruihong Yin, Yiqun Luo, Weiwei Wu, Shihong Pan, Tsung Yeh Hsieh, Xiao Huang, Yan Zhang, Sihan Li, Zirui Ren, Zengji Tu, Kyoung Whan Choe, Ke Sun, Vincenzo Galia, Vaisakhi Mishra, Jiashu He, Hongyang Wang, Zhiheng Yang, Shubham Gaur, Yifan Gu, Shuyi Wang, Zihao Zuo, Steven Dillmann, Yiyou Sun, Xinyang Han, Hanchung Lee, Molei Tao, Yue Zhao, Shilong Liu, Yuchen Wu, Xuandong Zhao, Xiangyi Li, Florian Brand, Gerbrand Ceder, Hartmut HaeffnerarXiv 2026 · arXiv

TL;DR

FrontierPhysics is a new, rigorous benchmark designed to evaluate AI agents on the full lifecycle of physics research—from literature review and experimental design to manuscript writing. The study reveals that while today’s best models possess vast physics knowledge, they fail at end-to-end research primarily due to a lack of rigorous reasoning and "expert intuition," with even the top-performing model passing only 27.4% of tasks.

What Problem Does This Paper Address?

Most existing AI benchmarks in science are "closed-ended." They ask models to solve isolated textbook problems, complete sub-tasks (like debugging a single script), or reproduce a paper where the solution is already provided. This doesn't reflect how science actually happens.

Real research is an "agentic" process: a Principal Investigator (PI) gives a high-level goal, and the researcher must navigate ambiguity, search the literature, design a plan, execute experiments in a sandbox, and synthesize findings into a publishable report. The authors argue that we lack a way to measure if AI can actually do physics research. FrontierPhysics fills this gap by providing 56 open-ended tasks based on the original, unpublished research of 42 practicing physicists.

Methods

The authors developed a pipeline to transform real-world research into standardized agent tasks.

  1. Task Design: Each task starts with a high-level prompt (the "PI's instructions") and a research proposal. The agent is placed in a sandbox environment with necessary data, simulators, and web access.
  2. The Workflow: Agents must investigate the topic, formulate a plan, execute code/simulations, and deliver two things: deterministic artifacts (numerical results) and a scientific manuscript (paper.pdf).
  3. Evaluation: The paper introduces a dual-grading system:
  4. Deterministic Verifiers: Automated scripts that check if the numerical outputs match reference values within a specific tolerance.
  5. Expert Rubrics: A "Reviewer Agent" (using a high-end LLM) grades the manuscript based on rubrics written by the original task authors.
  6. Blockers vs. Non-blockers: To pass, an agent must satisfy all "blocker" criteria (essential scientific validity, no reward hacking, no fabrication) and the deterministic verifier.
Overview of the FrontierPhysics pipeline, from task building to failure-mode analysis.

Datasets

The primary contribution is the FrontierPhysics benchmark, which includes:

  • 56 tasks across 9 domains (Condensed Matter, AMO Physics, Astrophysics, Fluid Dynamics, etc.).
  • 42 contributing physicists from 26 institutions.
  • A comparison against existing benchmarks (like CritPt, PRBench, and NatureBench), highlighting that FrontierPhysics is the only one that is fully agentic, uses original research, and grades both write-ups and results.

Key Insights and Ideas

  • Knowledge is not the Bottleneck: Surprisingly, agents rarely fail because they don't "know" the physics. Instead, the most common failure mode is a lack of rigorous reasoning (48%). This includes "circular self-verification," where an agent assumes its wrong answer is correct and builds a validation method that simply confirms its own error.
  • The "Intuition" Gap: 34% of failures were attributed to a lack of expert intuition. For example, an agent might fail to identify the primary source of error in an experiment (like Rydberg decay) because it lacks the "gut feeling" a seasoned experimentalist would have.
  • Instruction Following: Despite being "frontier" models, agents still struggled with basic constraints (26% failure rate), often missing specific requirements buried in the task description.
  • The Cost of Research: There is a clear Pareto frontier between cost/time and performance. However, for some top-tier models, increasing "inference effort" actually led to lower pass rates, suggesting that more "thinking time" can sometimes lead an agent further down a logical rabbit hole.
Analysis of why agents fail, showing that reasoning and intuition are the primary hurdles.

Key Results

The benchmark was tested across 22 models in 26 different configurations.

  • Claude Opus 5.5 emerged as the leader with a 27.4% pass rate.
  • GPT-6 Astra followed closely at 23.2%.
  • Many popular models, including Mistral Large 4 and Gemini 3.8 Flash, failed to pass a single task (0% pass rate) under the strict "all-blockers-must-pass" criteria.
  • Domain Variance: Pass rates varied wildly by field. Agents performed best in Fluid Dynamics (~50-60% pass rate) but struggled significantly in Statistical Physics and Chemical Physics.
Figure 1: FRONTIERPHYSICS LEADERBOARD. Overall pass rate (Pass@1) under the Table 2 configurations; a pass requires passing all verifier tests and blocker rubrics. Harnesses and inference level: GPT (Codex, max), Claude (Claude Code, max), Grok (Grok Build, xhigh), Gemini (Antigravity, high), Kimi (Claude Code, max), DeepSeek (Claude Code, max), MiMo (Claude Code, max), GLM (Claude Code, max), Mistral (Claude Code, max). Error bars show approximate 95% confidence intervals.

Limitations and Future Work

The authors acknowledge several limitations:

  • Sample Size: With only 56 tasks, the statistical significance of performance differences between specific physics sub-domains is limited.
  • Agent-as-Judge: While they validated the "Reviewer Agent" against human audits (finding high alignment), the evaluation still relies on an LLM to grade the qualitative aspects of the research.
  • Static Environment: The current sandbox is largely computational. Future iterations could involve more dynamic interactions with laboratory hardware or real-time data streams.
  • Human-AI Collaboration: The benchmark currently evaluates agents working in isolation. The authors suggest that future work should explore how these agents perform when collaborating with human researchers.

Summary updated

Abstract

AI agents are playing an increasingly important role in physics research. However, existing physics benchmarks only evaluate fixed questions, isolated sub-tasks, or paper reproduction with given solutions, and rarely check research quality alongside correctness. We introduce FrontierPhysics, an end-to-end benchmark with 56 tasks made by 42 physicists based on their own research, covering nine domains across theoretical, computational, and experimental physics. Each task requires agents to review the literature, develop a research plan, conduct research and experiments, and produce structured outputs and scientific manuscripts. Combining deterministic verifiers with expert-authored rubrics, FrontierPhysics evaluates whether agents' research is correct and publishable. Across 22 models in 26 configurations, no configuration passes more than 46 of its 168 rollouts. Manual review of agent trajectories shows that frontier agents' main weaknesses are insufficiently rigorous reasoning and limited expert intuition, rather than gaps in physics knowledge or execution errors.
BibTeX
@misc{you2026frontierphysics,
  author = {Bingran You and Wenjun Ke and Ria Rosenauer and Bo Liu and Wenbo Chen and Min Chen and Shuqi Xu and Yue Ma and Jiechao Feng and Bo Chen and Xiangkai Sun and Jiarui Liu and Jinen Guo and Zeyu Zhou and Xu Huang and Zhuohan Li and Haoyu Li and Heguang Lin and Lijie Ding and Yadong Zeng and Kangqi Zhang and Xinge Liu and Hanwen Liu and Zhenghong Li and Alexander von Recum and Yuchen Zhu and Felix Schwarzfischer and Yizhou Zhao and Jicheng Wang and Binxu Li and Qian Yu and Chan Li and Cheng Dai and Di Wang and Haoyu Zhang and Jiankai Sun and Kaituo Feng and Mian Qin and Ruihong Yin and Yiqun Luo and Weiwei Wu and Shihong Pan and Tsung Yeh Hsieh and Xiao Huang and Yan Zhang and Sihan Li and Zirui Ren and Zengji Tu and Kyoung Whan Choe and Ke Sun and Vincenzo Galia and Vaisakhi Mishra and Jiashu He and Hongyang Wang and Zhiheng Yang and Shubham Gaur and Yifan Gu and Shuyi Wang and Zihao Zuo and Steven Dillmann and Yiyou Sun and Xinyang Han and Hanchung Lee and Molei Tao and Yue Zhao and Shilong Liu and Yuchen Wu and Xuandong Zhao and Xiangyi Li and Florian Brand and Gerbrand Ceder and Hartmut Haeffner},
  title = {{FrontierPhysics: Evaluating Agents for End-to-End Physics Research}},
  year = {2026},
  eprint = {2611.06910},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06910},
}
Figure