OpenSeesAgentBench: A Benchmark and Evaluation Framework for Agentic Structural Analysis in OpenSeesPy

Takayuki ShinoharaAI4Science @ ICML 2026 · OpenReview

TL;DR

OpenSeesAgentBench is a new, multi-layered benchmark designed to evaluate how well AI agents handle structural engineering tasks using OpenSeesPy. Unlike previous benchmarks that only check if code is executable, this framework enforces "scientific contracts" to ensure AI-generated models are physically plausible, numerically accurate, and fully auditable.

What Problem Does This Paper Address?

In the world of earthquake engineering, precision is everything. A structural simulation that runs without crashing is useless if the damping is off, the time-stepping is unstable, or the model topology is slightly wrong.

Currently, Large Language Models (LLMs) are being tasked with generating these complex simulations. However, existing evaluation methods are often "system-centric"—they might tell you that an agent successfully generated a script, but they fail to distinguish between a script that simply executes and one that actually answers the engineering question correctly. There is a lack of standardized, auditable benchmarks that separate an agent's domain knowledge from its coding ability and its capacity to self-correct when a simulation fails.

Methods

The authors introduce a "contract-first" evaluation framework. Instead of letting the AI wander freely, the system uses a CaseSpec—a strict, typed definition (built with Pydantic) that specifies exactly what the model, materials, boundary conditions, and analysis settings should be.

The framework operates through a reference multi-agent system:

  1. Planner: Maps a natural-language request to a validated CaseSpec.
  2. Input Writer: Expands that spec into specific OpenSeesPy files.
  3. Runner: Executes the code in a "fail-closed" environment (meaning a run is only a success if it passes strict post-run validation).
  4. Reviewer: If things go wrong, a bounded repair loop uses a structured error taxonomy (like CONVERGENCE_FAIL or GEOM_TRANSF_ERROR) to attempt deterministic fixes.

Datasets

The paper introduces three distinct layers of evaluation:

  • OpenSeesQuery: 90 multiple-choice questions testing prerequisite domain knowledge (e.g., structural dynamics and API usage).
  • OpenSeesCodeBench: 24 tasks focused on prompt-to-code generation and numerical reasoning.
  • OpenSeesWorkflowBench: The core of the paper, featuring the OpenSeesBuildingBench calibration profile. This consists of 900 workflow cases sharded into Easy, Medium, and Hard categories (300 cases each). These cases are generated from eight different building archetypes (like 2D braced frames or 3D torsional buildings) and include variations in linguistic phrasing to test the agent's robustness.

Key Insights and Ideas

  • Executability $\neq$ Correctness: The authors found a significant gap between "completed rate" (code that ran) and "expected match rate" (code that actually met the scientific contract). On harder tasks, many agents produced runnable code that was scientifically invalid.
  • The Power of the "Scientific Contract": By using the CaseSpec as an intermediate state, the benchmark makes the AI's "intent" auditable. You can see exactly where the agent's plan diverged from the engineering requirements before a single line of code is even run.
  • Structured Failure is Useful: Instead of generic error messages, the benchmark uses a specific taxonomy of structural engineering failures. This allows for "bounded repair," where the agent can attempt to fix specific numerical issues rather than just guessing.

Key Results

The study served as a "calibration" to ensure the benchmark's difficulty levels were set correctly. Using a reference agent stack, the observed success rates (Expected Match Rates) tracked the intended targets almost perfectly:

  • Easy Shard: 89.33% (Target: 89.67%)
  • Medium Shard: 70.33% (Target: 70.33%)
  • Hard Shard: 60.67% (Target: 60.67%)
  • Overall: 73.44% across all 900 cases.

These results confirm that the benchmark provides a stable, reproducible ladder of difficulty for future AI agents to climb.

Limitations and Future Work

The authors are transparent about the current scope:

  • Narrow Analysis Type: The current version is centered on linear transient (time-history) analysis. Future versions will need to tackle nonlinear behavior and more complex structural elements.
  • No Head-to-Head Yet: This paper focuses on the benchmark and the reference stack. It does not yet provide a comparative ranking of different LLM architectures (like GPT-4 vs. Claude 3.5).
  • Local Execution: The current setup is designed for local calibration; scaling this to massive, cloud-based "AI Scientist" workflows is a logical next step.

The ultimate goal is to move toward a future where AI agents don't just write code, but act as reliable, auditable partners in high-stakes engineering design.

Summary updated

Abstract

Large language model (LLM) agents are being applied to structural analysis, but evaluation is often reduced to executability and can blur benchmark construction with agent capability. We present OpenSeesAgentBench, an OpenSeesPy benchmark and executable reference evaluation stack spanning OpenSeesQuery (90 questions), OpenSeesCodeBench (24 code tasks), and OpenSeesWorkflowBench via the OpenSeesBuildingBench calibration profile (900 workflow cases). The workflow layer uses contract-first evaluation with strict CaseSpec validation, fail-closed checks, and bounded repair. In the current paper, we report only the reference stack, so the workflow results should be read as a calibration study rather than a head-to-head systems comparison: observed shard expected-match rates of 89.33\%, 70.33\%, and 60.67\% stay within one case of the profile targets. These results show that the released workflow profile preserves its intended difficulty under executable expansion and provides an auditable benchmark for future comparative evaluation; they do not, by themselves, establish superiority of a particular agent architecture.
BibTeX
@inproceedings{shinohara2026openseesagentbench,
  author = {Takayuki Shinohara},
  title = {{OpenSeesAgentBench: A Benchmark and Evaluation Framework for Agentic Structural Analysis in OpenSeesPy}},
  booktitle = {AI4Science @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=J263rcTa09},
}
Figure