TL;DR
AgentScript-Eval is a new execution-graded benchmark designed to test LLMs on a proprietary, enterprise-specific domain-specific language (DSL) used by Salesforce. The study reveals that while frontier models excel at public languages like Python, they struggle significantly with "private" DSLs, with even the best models failing to author code from scratch zero-shot. The biggest performance gains come not from increasing model scale, but from implementing agentic "compile-and-edit" loops and optimizing how domain guidance is structured for the model.
What Problem Does This Paper Address?
Most code-generation benchmarks (like HumanEval or SWE-bench) focus on languages that are ubiquitous in public pretraining data, such as Python and Java. Because frontier LLMs have seen billions of lines of this code, their high performance might reflect "memorization" or familiarity rather than a general ability to reason about programming logic.
In the enterprise world, however, much of the logic is written in proprietary Domain-Specific Languages (DSLs). These languages are often declarative, highly specialized, and—crucially—absent from public training sets. This paper investigates whether LLM coding capabilities actually transfer to these "low-resource" enterprise environments. To do this, they introduce AgentScript, the DSL used in Salesforce’s "Agentforce" platform for building conversational agents.
Methods
The authors developed an evaluation harness (shown in Figure 1) that goes beyond simple text similarity. Instead, it uses execution-graded scoring:
- Generate: The model produces AgentScript code (either in a single shot or through an agentic loop).
- Compile: The code is checked for syntax and structural validity.
- Execute: If it compiles, the agent is run against a "user simulator" in multi-turn dialogues with stubbed backends.
- Score: Success is measured by whether the agent correctly invokes actions and reaches the expected state.
The study compared five models (including frontier models like GPT-5.5 and Claude Opus 4.8, and open-weight models like Gemma-4 31B) across different configurations: zero-shot, two-shot, and various "agentic" setups where the model can see compiler errors and test results to fix its own code.
Datasets
The paper introduces AgentScript-Eval, a benchmark consisting of 100 real-world tasks:
- Authoring (30 tasks): Creating an agent from a natural-language spec.
- Debugging (40 tasks): Fixing an uncompiling script. These tasks include a variety of defect classes like reasoning lifecycle errors and DSL syntax issues (see Table 2).
- Optimization (30 tasks): Improving a working agent to meet a new objective.
The benchmark includes 630 executable test cases, ensuring that "passing" requires functional correctness, not just plausible-looking code.
Key Insights and Ideas
- The Zero-Shot Wall: Every model tested scored 0% on authoring tasks from scratch in a zero-shot setting. This highlights a massive gap in LLM "reasoning" when the syntax isn't already baked into the model's weights.
- Scale is a Weak Predictor: Surprisingly, the 31B Gemma-4 model achieved the highest compilation rate (48%) among direct-generation models, outperforming much larger hosted models while being significantly cheaper.
- The "Agentic" Lever: The single most effective way to improve performance was giving the model a compiler. Allowing a model to see its own errors and iterate (an agentic loop) roughly doubled partial-credit accuracy.
- Guidance Density Matters: The authors found that how you explain the DSL to the model is more important than how you deliver it. A "v3" skill pack that used "progressive disclosure" (providing a lean entry point that pulls in details only when needed) outperformed a massive, all-at-once documentation dump by 23 points.
Key Results
The results (summarized in Table 1) show that enterprise DSL generation is a "hard" regime:
- Best Zero-Shot Performance: Claude Opus 4.8 reached only 25.0% pass@3.
- Compilation vs. Correctness: Many models could produce code that compiled but failed to behave correctly. For example, Gemma-4 compiled 48% of the time but only solved 15% of tasks.
- Top Performance: The best configuration (Agentic Opus 4.8 with "v3" rewritten guidance) reached a 37.0% pass@1 and a 98.0% compilation rate.
Limitations and Future Work
The authors note that while agentic loops solved the "surface-form" difficulty (getting the code to compile), behavioral correctness remains the bottleneck. Even with compiler feedback and graded results, the best configuration still failed 40% of tasks at a partial-credit level.
Future work suggests moving away from "broad regeneration" (asking the model to try again) toward "targeted repair," where the agent uses specific failure signals to pinpoint and fix logic errors. The study also suggests that teams building AI agents should treat "guidance density" as a primary design variable rather than just dumping documentation into a prompt.
Summary updated