TL;DR
OpenSeesAgentBench is a new, multi-layered benchmark designed to evaluate how well AI agents handle structural engineering tasks using OpenSeesPy. Unlike previous benchmarks that only check if code is executable, this framework enforces "scientific contracts" to ensure AI-generated models are physically plausible, numerically accurate, and fully auditable.
What Problem Does This Paper Address?
In the world of earthquake engineering, precision is everything. A structural simulation that runs without crashing is useless if the damping is off, the time-stepping is unstable, or the model topology is slightly wrong.
Currently, Large Language Models (LLMs) are being tasked with generating these complex simulations. However, existing evaluation methods are often "system-centric"—they might tell you that an agent successfully generated a script, but they fail to distinguish between a script that simply executes and one that actually answers the engineering question correctly. There is a lack of standardized, auditable benchmarks that separate an agent's domain knowledge from its coding ability and its capacity to self-correct when a simulation fails.
Methods
The authors introduce a "contract-first" evaluation framework. Instead of letting the AI wander freely, the system uses a CaseSpec—a strict, typed definition (built with Pydantic) that specifies exactly what the model, materials, boundary conditions, and analysis settings should be.
The framework operates through a reference multi-agent system:
- Planner: Maps a natural-language request to a validated
CaseSpec. - Input Writer: Expands that spec into specific OpenSeesPy files.
- Runner: Executes the code in a "fail-closed" environment (meaning a run is only a success if it passes strict post-run validation).
- Reviewer: If things go wrong, a bounded repair loop uses a structured error taxonomy (like
CONVERGENCE_FAILorGEOM_TRANSF_ERROR) to attempt deterministic fixes.
Datasets
The paper introduces three distinct layers of evaluation:
- OpenSeesQuery: 90 multiple-choice questions testing prerequisite domain knowledge (e.g., structural dynamics and API usage).
- OpenSeesCodeBench: 24 tasks focused on prompt-to-code generation and numerical reasoning.
- OpenSeesWorkflowBench: The core of the paper, featuring the OpenSeesBuildingBench calibration profile. This consists of 900 workflow cases sharded into Easy, Medium, and Hard categories (300 cases each). These cases are generated from eight different building archetypes (like 2D braced frames or 3D torsional buildings) and include variations in linguistic phrasing to test the agent's robustness.
Key Insights and Ideas
- Executability $\neq$ Correctness: The authors found a significant gap between "completed rate" (code that ran) and "expected match rate" (code that actually met the scientific contract). On harder tasks, many agents produced runnable code that was scientifically invalid.
- The Power of the "Scientific Contract": By using the
CaseSpecas an intermediate state, the benchmark makes the AI's "intent" auditable. You can see exactly where the agent's plan diverged from the engineering requirements before a single line of code is even run. - Structured Failure is Useful: Instead of generic error messages, the benchmark uses a specific taxonomy of structural engineering failures. This allows for "bounded repair," where the agent can attempt to fix specific numerical issues rather than just guessing.
Key Results
The study served as a "calibration" to ensure the benchmark's difficulty levels were set correctly. Using a reference agent stack, the observed success rates (Expected Match Rates) tracked the intended targets almost perfectly:
- Easy Shard: 89.33% (Target: 89.67%)
- Medium Shard: 70.33% (Target: 70.33%)
- Hard Shard: 60.67% (Target: 60.67%)
- Overall: 73.44% across all 900 cases.
These results confirm that the benchmark provides a stable, reproducible ladder of difficulty for future AI agents to climb.
Limitations and Future Work
The authors are transparent about the current scope:
- Narrow Analysis Type: The current version is centered on linear transient (time-history) analysis. Future versions will need to tackle nonlinear behavior and more complex structural elements.
- No Head-to-Head Yet: This paper focuses on the benchmark and the reference stack. It does not yet provide a comparative ranking of different LLM architectures (like GPT-4 vs. Claude 3.5).
- Local Execution: The current setup is designed for local calibration; scaling this to massive, cloud-based "AI Scientist" workflows is a logical next step.
The ultimate goal is to move toward a future where AI agents don't just write code, but act as reliable, auditable partners in high-stakes engineering design.
Summary updated