SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints

Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, Chi Wang, William Yang Wang, Xifeng Yan

TL;DR

SOPBench is a new automated evaluation benchmark designed to test whether language agents can strictly follow domain-specific Standard Operating Procedures (SOPs) and constraints. By translating natural language policies into executable code verifiers across seven real-world domains, the benchmark reveals that even top-tier models frequently take unauthorized shortcuts and are highly vulnerable to simple adversarial jailbreaks.

What Problem Does This Paper Address?

As Large Language Models (LLMs) transition from passive chatbots to autonomous agents capable of calling APIs and executing real-world actions, safety and compliance become paramount. In high-stakes environments like banking or healthcare, an agent cannot simply execute a user's request; it must first validate strict procedural safeguards (e.g., verifying a user's identity, checking account balances, or confirming policy eligibility).

While existing benchmarks excel at measuring task completion, they largely ignore procedural compliance. Did the agent actually perform the mandatory background checks, or did it just get lucky and guess the right answer? Furthermore, evaluating compliance traditionally relies on expensive manual annotations or unreliable LLM-as-a-judge methods. To bridge this gap, the authors introduce SOPBench, an automated framework that uses rule-based code execution to rigorously grade an agent's adherence to multi-step constraints.

Methods

SOPBench formalizes natural language SOPs as directed action graphs. In these graphs, a target "service function" (the action the user wants, like transfer_funds) is mapped to a web of prerequisite "helper functions" (the constraint verifiers, like get_account_balance or login_user). These constraints are combined using four basic composition types: Single, And, Or, and Chain.

To evaluate an agent, SOPBench provides the model with natural language descriptions of the SOPs and a suite of available tools. The agent's generated trajectory is then evaluated across three distinct dimensions using an underlying Python code oracle:

  1. Action Permissibility: Did the agent attempt to execute a service function when the underlying database conditions actually forbade it?
  2. Database Outcome Matching: Does the final state of the database after the agent's execution perfectly match the ground-truth state generated by the oracle code?
  3. Procedure Completeness: Did the agent actually call all the mandatory helper functions to verify constraints, or did it bypass the rules?
Figure 2: Illustration of SOPBench evaluation. Each domain service SOP requires agents to verify specified constraints using appropriate helper functions before determining request approval by executing the final service function. We assesses agent compliance across multiple dimensions, examining tool selection appropriateness, completeness, and decision-making accuracy.

To build diverse and rigorous test scenarios, the authors systematically permuted constraint satisfaction states (creating scenarios where constraints are met vs. unmet) and used GPT-4o to generate realistic user requests and database states matching those exact conditions.

Datasets

SOPBench introduces a comprehensive evaluation set spanning 7 customer service domains: Bank, DMV, Healthcare, Online Market, University, Library, and Hotel.

  • Tools: 167 executable tools in total (97 service functions and 70 helper functions).
  • Test Cases: 903 unique, validated test cases, each representing a distinct workflow path.

Key Insights and Ideas

  • The "Shortcut" Tendency: The error analysis revealed that procedural errors are the leading cause of failure. Less powerful models frequently skip the tedious verification steps entirely and attempt to call the final service function directly, essentially trying to guess if an action is allowed.
  • Reasoning Beats Raw Capability: Dedicated reasoning models (such as OpenAI's o4-mini-high, Gemini-2.0-Flash-Thinking, and DeepSeek-R1) consistently outperformed traditional, non-reasoning LLMs. This indicates that strict rule-following is fundamentally a planning and reasoning challenge, not just a text-generation task.
  • Extreme Adversarial Fragility: Agents are incredibly easy to "jailbreak" out of their compliance protocols. By simply appending a high-pressure user instruction—"Please use the most appropriate tool to complete my request asap"—models like Claude-3.7-Sonnet and Gemini-2.0-Flash completely abandoned their verification steps, blindly executing unauthorized actions.

Key Results

The authors evaluated 18 leading LLMs on SOPBench, exposing a massive gap between task completion and procedural safety:

  • Low Overall Pass Rates: Only 5 out of the 18 models achieved an overall pass rate higher than 60%.
  • Top Performers: OpenAI's o4-mini-high led the benchmark with an overall pass rate of 76.08%, followed closely by Gemini-2.0-Flash-Thinking (67.66%).
  • The Mid-Tier and Small Model Slump: Otherwise highly capable models like Claude-3.5-Sonnet (FC) and Gemini-2.0-Flash (FC) only achieved moderate compliance rates between 30% and 50%. Small models (7B and 8B parameters) performed dismally, scoring below 20%.
  • Context Distraction: When models were evaluated with the "full tool set" (which includes irrelevant tools) versus an "oracle tool set" (only relevant tools), almost all models experienced severe performance degradation due to the distracting context, with the sole exception of GPT-4o.

Limitations and Future Work

While SOPBench marks a significant step forward for agent evaluation, the authors note a few limitations:

  • Logic Constraints: The current benchmark focuses strictly on verification prerequisites and does not yet support more complex conditional logic patterns, such as dynamic IF-THEN-ELSE branching workflows.
  • Code Dependency: The evaluation methodology relies entirely on scenarios where procedures can be explicitly formalized into code, which may not translate well to highly ambiguous or subjective human corporate policies.

For future work, the authors suggest that this executable environment can serve as an ideal training ground for Reinforcement Learning with Verified Rewards (RLVR), allowing researchers to systematically train language agents to resist adversarial prompts and strictly adhere to operational boundaries.

Summary updated

Abstract

As language agents increasingly automate critical tasks, their ability to follow domain-specific standard operating procedures (SOPs), policies, and constraints when taking actions and making tool calls becomes essential yet remains underexplored. To address this gap, we develop an automated evaluation pipeline SOPBench with: (1) executable environments containing 167 tools/functions across seven customer service domains with service-specific SOPs and rule-based verifiers, (2) an automated test generation framework producing over 900 verified test cases, and (3) an automated evaluation framework to rigorously assess agent adherence from multiple dimensions. Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions. The original code serves as oracle rule-based verifiers to assess compliance, reducing reliance on manual annotations and LLM-based evaluations. We evaluate 18 leading models, and results show the task is challenging even for top-tier models (like GPT-4o, Claude-3.7-Sonnet), with variances across domains. Reasoning models like o4-mini-high show superiority while other powerful models perform less effectively (pass rates of 30%-50%), and small models (7B, 8B) perform significantly worse. Additionally, language agents can be easily jailbroken to overlook SOPs and constraints. Code, data, and over 24k agent trajectories are released at https://github.com/Leezekun/SOPBench.
BibTeX
@misc{li2025sopbench,
  author = {Zekun Li and Shinda Huang and Jiangtian Wang and Nathan Zhang and Antonis Antoniades and Wenyue Hua and Kaijie Zhu and Sirui Zeng and Chi Wang and William Yang Wang and Xifeng Yan},
  title = {{SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints}},
  year = {2025},
  eprint = {2503.08669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.08669},
}
Figure