AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL

Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. MuiarXiv 2026 · arXiv

TL;DR

AgentScript-Eval is a new execution-graded benchmark designed to test LLMs on a proprietary, enterprise-specific domain-specific language (DSL) used by Salesforce. The study reveals that while frontier models excel at public languages like Python, they struggle significantly with "private" DSLs, with even the best models failing to author code from scratch zero-shot. The biggest performance gains come not from increasing model scale, but from implementing agentic "compile-and-edit" loops and optimizing how domain guidance is structured for the model.

What Problem Does This Paper Address?

Most code-generation benchmarks (like HumanEval or SWE-bench) focus on languages that are ubiquitous in public pretraining data, such as Python and Java. Because frontier LLMs have seen billions of lines of this code, their high performance might reflect "memorization" or familiarity rather than a general ability to reason about programming logic.

In the enterprise world, however, much of the logic is written in proprietary Domain-Specific Languages (DSLs). These languages are often declarative, highly specialized, and—crucially—absent from public training sets. This paper investigates whether LLM coding capabilities actually transfer to these "low-resource" enterprise environments. To do this, they introduce AgentScript, the DSL used in Salesforce’s "Agentforce" platform for building conversational agents.

Methods

The authors developed an evaluation harness (shown in Figure 1) that goes beyond simple text similarity. Instead, it uses execution-graded scoring:

  1. Generate: The model produces AgentScript code (either in a single shot or through an agentic loop).
  2. Compile: The code is checked for syntax and structural validity.
  3. Execute: If it compiles, the agent is run against a "user simulator" in multi-turn dialogues with stubbed backends.
  4. Score: Success is measured by whether the agent correctly invokes actions and reaches the expected state.
Figure 1: The evaluation harness. All configurations share the same tasks, environment, and checks, differing only in how the program is produced

The study compared five models (including frontier models like GPT-5.5 and Claude Opus 4.8, and open-weight models like Gemma-4 31B) across different configurations: zero-shot, two-shot, and various "agentic" setups where the model can see compiler errors and test results to fix its own code.

Datasets

The paper introduces AgentScript-Eval, a benchmark consisting of 100 real-world tasks:

  • Authoring (30 tasks): Creating an agent from a natural-language spec.
  • Debugging (40 tasks): Fixing an uncompiling script. These tasks include a variety of defect classes like reasoning lifecycle errors and DSL syntax issues (see Table 2).
  • Optimization (30 tasks): Improving a working agent to meet a new objective.
Table 2: Defect classes injected into debugging tasks, with the number of tasks containing at least one defect of that class. Counts exceed 40 because tasks carry multiple defects.

The benchmark includes 630 executable test cases, ensuring that "passing" requires functional correctness, not just plausible-looking code.

Key Insights and Ideas

  • The Zero-Shot Wall: Every model tested scored 0% on authoring tasks from scratch in a zero-shot setting. This highlights a massive gap in LLM "reasoning" when the syntax isn't already baked into the model's weights.
  • Scale is a Weak Predictor: Surprisingly, the 31B Gemma-4 model achieved the highest compilation rate (48%) among direct-generation models, outperforming much larger hosted models while being significantly cheaper.
  • The "Agentic" Lever: The single most effective way to improve performance was giving the model a compiler. Allowing a model to see its own errors and iterate (an agentic loop) roughly doubled partial-credit accuracy.
  • Guidance Density Matters: The authors found that how you explain the DSL to the model is more important than how you deliver it. A "v3" skill pack that used "progressive disclosure" (providing a lean entry point that pulls in details only when needed) outperformed a massive, all-at-once documentation dump by 23 points.

Key Results

The results (summarized in Table 1) show that enterprise DSL generation is a "hard" regime:

  • Best Zero-Shot Performance: Claude Opus 4.8 reached only 25.0% pass@3.
  • Compilation vs. Correctness: Many models could produce code that compiled but failed to behave correctly. For example, Gemma-4 compiled 48% of the time but only solved 15% of tasks.
  • Top Performance: The best configuration (Agentic Opus 4.8 with "v3" rewritten guidance) reached a 37.0% pass@1 and a 98.0% compilation rate.
Table 1: Pass rate is mean per-task partial credit; tokens are mean output per task

Limitations and Future Work

The authors note that while agentic loops solved the "surface-form" difficulty (getting the code to compile), behavioral correctness remains the bottleneck. Even with compiler feedback and graded results, the best configuration still failed 40% of tasks at a partial-credit level.

Future work suggests moving away from "broad regeneration" (asking the model to try again) toward "targeted repair," where the agent uses specific failure signals to pinpoint and fix logic errors. The study also suggests that teams building AI agents should treat "guidance density" as a primary design variable rather than just dumping documentation into a prompt.

Summary updated

Abstract

Domain-specific languages (DSLs) are widely used in enterprise software, giving a compact interface to complex tasks, from data access to infrastructure automation. Platforms increasingly extend them to agents, where a developer declares the topics, permitted actions, and guardrails of an agent rather than writing general-purpose code. Large language models (LLMs) write general-purpose code well, but that ability has been measured almost entirely on languages abundant in public pretraining data, and enterprise DSLs are neither abundant nor public. We introduce AgentScript-Eval, an execution-graded benchmark of 100 real tasks for AgentScript, the declarative DSL that Salesforce uses in Agentforce to build conversational agents. Tasks span authoring an agent from a specification, debugging a broken definition, and optimizing a working one. Evaluating five models zero-shot, we find that the regime is hard: the best model reaches only 25.0% pass@3, and every model scores 0% on authoring from scratch. We further find that model scale and price are weak predictors of success -- a 31B open-weight model attains the highest compilation rate of any model tested (48.0%) and matches the strongest frontier model on partial-credit accuracy at the lowest token cost. Adding an agentic compile-and-edit loop is the single largest lever, roughly doubling partial-credit accuracy. Beyond that gain, the wording of domain guidance matters more than its packaging. The same skill pack scores 28% as plain documentation but only 14% as a live plugin. Rewriting one skill inside that plugin then raises it to 37%, our best result, with catalog and tooling untouched. Larger models did not reliably outperform smaller ones; what mattered was the harness surrounding the model and the guidance supplied to it.
BibTeX
@misc{khati2026agentscript,
  author = {Dipin Khati and Shubham Mehrotra and Bin Bi and Zhou Yu and Denys Poshyvanyk and James Zhu and Sitaram Asur and Phil L. Mui},
  title = {{AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL}},
  year = {2026},
  eprint = {2611.13183},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13183},
}
Figure