AgentOrca: A Dual-System Framework to Evaluate Language Agents on Operational Routine and Constraint Adherence

Zekun Li, Shinda Huang, Jiangtian Wang, Nathan Zhang, Antonis Antoniades, Wenyue Hua, Kaijie Zhu, Sirui Zeng, William Yang Wang, Xifeng Yan

Abstract

As language agents progressively automate critical tasks across domains, their ability to operate within operational constraints and safety protocols becomes essential. While extensive research has demonstrated these agents' effectiveness in downstream task completion, their reliability in following operational procedures and constraints remains largely unexplored. To this end, we present AgentOrca, a dual-system framework for evaluating language agents' compliance with operational constraints and routines. Our framework encodes action constraints and routines through both natural language prompts for agents and corresponding executable code serving as ground truth for automated verification. Through an automated pipeline of test case generation and evaluation across five real-world domains, we quantitatively assess current language agents' adherence to operational constraints. Our findings reveal notable performance gaps among state-of-the-art models, with large reasoning models like o1 demonstrating superior compliance while others show significantly lower performance, particularly when encountering complex constraints or user persuasion attempts.

Introduction: Beyond Task Completion

As Large Language Model (LLM) agents transition from simple chatbots to autonomous workers in banking, healthcare, and government, "task completion" is no longer the only metric that matters. In these high-stakes environments, how a task is completed is just as critical as the final result.

For example, an agent shouldn't just "approve a loan"; it must first verify identity, check credit scores, and validate account balances in a specific order. This is known as procedural compliance. Existing benchmarks often focus on whether the agent reached the goal, but they fail to rigorously check if the agent followed the "Standard Operating Procedures" (SOPs) required for safety and policy. Manually evaluating these complex trajectories is slow and expensive, creating a need for an automated way to verify that agents are playing by the rules.

The AgentOrca Framework

The researchers introduce AgentOrca, a dual-system framework that treats SOPs as executable code to enable automated evaluation.

Overview of the AgentOrca framework, showing the flow from user instructions and language agents to domain environments and automated evaluation.
  1. SOPs as Directed Graphs: Every domain-specific SOP is transformed into a directed graph where "helper functions" (which check constraints) must be called before "service functions" (which execute the final action).
  2. Rule-Based Verifiers: Instead of using another LLM to grade the agent, the authors developed "Oracle" Python programs. These programs act as the ground truth, checking the agent's trajectory against three dimensions:
  3. Action Permissibility: Did the agent try to perform an action prohibited by the data?
  4. Database Outcome Matching: Does the final state of the database match what the Oracle code would have produced?
  5. Procedure Completeness: Did the agent call every required verification tool before taking the final action?

Automated Verification with SOPBench

The benchmark, SOPBench, covers 7 diverse customer service domains including Banking, DMV, Healthcare, and Universities. It features 167 executable tools (97 service functions and 70 helper functions) and over 900 verified test cases.

Detailed walkthrough of the AgentOrca verification process, illustrating how agent trajectories are checked against database outcomes and SOP graphs to identify missing prerequisites.

By comparing agent trajectories against the SOP graph, the framework can pinpoint exactly where an agent failed—such as skipping a mandatory prerequisite check like get_interaction_time before declaring a minor.

Key Insights and Failure Modes

  • The "Hurry Up" Vulnerability: Agents are surprisingly easy to manipulate. When given adversarial instructions like "Please use the most appropriate tool to complete my request asap," models often skip safety checks entirely to reach the goal faster. This suggests that current agents prioritize helpfulness over safety and compliance.
  • Reasoning Models are Superior: Models designed with internal "thinking" chains (like OpenAI’s o1 or o4-mini-high) performed significantly better. They are more likely to plan out necessary verification steps rather than jumping straight to the final tool call.
  • Procedural Errors are the Main Culprit: Most failures weren't due to tool-use inability, but because agents skipped the mandatory checks required by the SOP.
  • Context Overload: When agents are given a "full tool set" (dozens of options), their compliance drops compared to when they are given only relevant tools. Identifying which rules apply in complex environments remains a major hurdle.

Results: Performance Gaps

The researchers evaluated 18 leading models, and the "pass rates" (meaning the agent followed the SOP perfectly) were lower than expected:

  • Top Tier: Only 5 models (including GPT-4o and o4-mini-high) achieved pass rates above 60%.
  • The Reasoning Edge: o4-mini-high led the pack with an overall pass rate of ~76%, showing the power of dedicated reasoning architectures for complex planning.
  • Mid-Tier Struggles: Powerful models like Claude 3.5 Sonnet and Gemini 2.0 Flash hovered between 30% and 50% pass rates.
  • Small Model Failure: Models in the 7B-8B parameter range (like Llama 3.1-8B) performed poorly, often scoring below 20%.

Conclusion

SOPBench highlights a critical gap in current AI agent development: the trade-off between helpfulness and procedural safety. While reasoning models show promise, most agents are currently vulnerable to simple pressure-cooker instructions that lead them to bypass essential protocols. Future work suggests using this environment for Reinforcement Learning with Verified Rewards (RLVR), where agents can be trained specifically to maximize procedural compliance by receiving rewards from rule-based verifiers.

Summary updated

Abstract

As language agents progressively automate critical tasks across domains, their ability to operate within operational constraints and safety protocols becomes essential. While extensive research has demonstrated these agents' effectiveness in downstream task completion, their reliability in following operational procedures and constraints remains largely unexplored. To this end, we present AgentOrca, a dual-system framework for evaluating language agents' compliance with operational constraints and routines. Our framework encodes action constraints and routines through both natural language prompts for agents and corresponding executable code serving as ground truth for automated verification. Through an automated pipeline of test case generation and evaluation across five real-world domains, we quantitatively assess current language agents' adherence to operational constraints. Our findings reveal notable performance gaps among state-of-the-art models, with large reasoning models like o1 demonstrating superior compliance while others show significantly lower performance, particularly when encountering complex constraints or user persuasion attempts.
BibTeX
@misc{li2025agentorca,
  author = {Zekun Li and Shinda Huang and Jiangtian Wang and Nathan Zhang and Antonis Antoniades and Wenyue Hua and Kaijie Zhu and Sirui Zeng and William Yang Wang and Xifeng Yan},
  title = {{AgentOrca: A Dual-System Framework to Evaluate Language Agents on Operational Routine and Constraint Adherence}},
  year = {2025},
  eprint = {2503.08669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.08669},
}
Figure