TL;DR
SeisAgentBench is a new, failure-aware benchmark designed to evaluate how AI agents orchestrate complex earthquake rupture analysis. By moving beyond simple input-output pairs to "interaction episodes," it tests whether AI can not only use scientific tools but also handle the inevitable disagreements and uncertainties that arise when modeling the Earth's crust.
What Problem Does This Paper Address?
Seismology is a "fragmented" field. To analyze an earthquake, a researcher must chain together disparate tools for data preprocessing (ObsPy), source inversion (Grond, WISP), and high-fidelity physics simulations (SPECFEM, SW4). Each tool has its own data formats, assumptions, and "hidden" defaults.
For AI agents to be useful in this space, they need to do more than just call an API. Earthquake rupture inversion is notoriously "non-unique"—multiple different models can explain the same data. A naive AI might pick the first result it gets, whereas a scientist would validate that result against a physics simulation. Currently, there is no standardized way to evaluate if an AI agent is actually performing this kind of "scientific reasoning" or just getting lucky.
Methods
The authors introduce several novel concepts to move AI evaluation from "chatbot" to "scientist":
- Interaction Episodes: Instead of a single prompt, the benchmark unit is an "episode"—a complete case file recording the scientific intent, the data used, every backend action taken, and how the final conclusion was reached.
- Shared Object Layer: To bridge the gap between tools, the paper defines a standardized schema (e.g.,
EventHypothesis,WaveformBundle). This acts as a "contract" that ensures scientific meaning is preserved as data moves from an inversion tool to a 3D wave simulator. - Multi-Agent Orchestration: The framework assumes a hierarchy where a Supervisor Agent routes tasks to specialized agents (Data, Inversion, Validation).
- Failure-Awareness: The benchmark explicitly includes scenarios where tools disagree. For example, an inversion might produce a model that fits the data but is physically implausible when run through a forward simulation. The benchmark evaluates if the agent can detect this conflict and "rerank" or "abstain" rather than providing a wrong answer.
Datasets
SeisAgentBench is rolled out in three phases:
- Phase 1 (Synthetic): Controlled environments where the "ground truth" is known, allowing for high-quality early testing.
- Phase 2 (Historical): Replayable episodes based on real-world historical earthquakes and published models, introducing realistic ambiguity.
- Phase 3 (Failure-Heavy): Scenarios specifically designed to break the workflow—conflicting outputs, diverging preprocessing choices, or simulation rejections.
The data includes strong-motion records, teleseismic data (distant recordings), and GNSS (ground displacement) measurements.
Key Insights and Ideas
- Orchestration is the Bottleneck: The authors argue that scientific progress in AI-for-Science isn't just about better LLMs or faster solvers, but about the interoperable workflows that connect them.
- Disagreement as a Signal: In many AI benchmarks, a "failure" is a bug. In SeisAgentBench, a "failure" (like two tools disagreeing) is a critical piece of scientific evidence. High-quality agents are expected to treat disagreement as a signal to investigate further.
- Physics-First, LLM-Second: While the benchmark includes "GeoGPT" as a supplemental reviewer, it is explicitly not the ground truth. Deterministic physics solvers remain the ultimate authority for validation.
Key Results
While the paper focuses on the design of the benchmark rather than the performance of a specific model, it establishes a rigorous evaluation protocol based on three dimensions:
- Scientific Adequacy: Does the model fit the physics? (Measured via waveform misfit and moment consistency).
- Workflow Competence: Did the agent follow the rules? (Measured via schema validity and provenance completeness).
- Failure-Aware Reasoning: Did the agent catch the error? (Measured by the agent's ability to rerank or escalate when backends disagree).
The paper also defines Reference Baselines, ranging from simple "single-tool" runs to "oracle-assisted" agents, to help researchers isolate whether their AI is improving because of better scientific modeling or better orchestration.
Limitations and Future Work
- Narrow Initial Scope: The current release focuses strictly on rupture inversion and forward validation. Future versions (v2+) aim to extend the workflow to hazard estimation and regional impact modeling (using tools like OpenQuake).
- Computational Cost: High-fidelity 3D simulations (like those in SeisSol or SW4) are computationally expensive. The authors acknowledge that not all users will be able to run the most demanding "challenge" episodes.
- The "Ground Truth" Problem: For real-world events, there is rarely a single "correct" answer. The benchmark handles this through comparative labeling, but it remains a more complex evaluation than standard machine learning tasks.
Summary updated