SHERPA: A Model-Driven Framework for Large Language Model Execution

Boqi Chen, Kua Chen, José Antonio Hernández López, Gunter Mussbacher, Dániel Varró, Amir Feizpour

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, their stochastic nature often leads to challenges in maintaining long-term consistency and adhering to complex, multi-step workflows. SHERPA (Stateful Hierarchical Execution and Reasoning for Process Automation) is a novel model-driven framework designed to address these limitations by structuring LLM execution through hierarchical state machines. By explicitly embedding domain-specific workflows and human best practices into these state machines, SHERPA improves performance on complex tasks while providing a mechanism to reduce computational overhead.

The Challenge of Structured Reasoning

While techniques like Chain-of-Thought (CoT) prompting have improved LLM reasoning, they do not provide a rigorous method for enforcing domain-specific "best practices." For instance, a software engineer follows a specific sequence of steps to validate code, a process that is difficult to guarantee through simple prompting alone. Furthermore, many existing agentic frameworks hard-code their workflows, which limits flexibility and makes it difficult for researchers to iterate on logic structures without significant code rewrites. SHERPA addresses this gap by decoupling workflow logic from implementation, treating the execution process as a model that can be independently designed and optimized.

The SHERPA Framework

SHERPA bridges the gap between Model-Driven Engineering (MDE) and LLM execution through an architecture composed of three core components: the Hierarchical State Machine, the Policy Engine, and the Belief System.

The SHERPA architecture, illustrating the interaction between the State Machine, Policy Engine, and Belief System.
  1. Hierarchical State Machine (SM): Tasks are decomposed into states (simple or composite). Transitions are triggered by events, such as user inputs or LLM outputs. This allows for nested logic, such as an "Inspection" state that can loop back to a "Generation" state upon detecting errors.
  2. Policy Engine: This component determines the next transition. SHERPA supports a hybrid policy: deterministic rule-based logic for straightforward paths and LLM-based decision-making for complex choices. A "fast-forward" mechanism allows the system to skip expensive LLM calls when only one valid transition exists.
  3. Belief System: This serves as a structured memory, tracking the "trajectory" (the path through the state machine) and an "execution log" (detailed inputs and outputs of every action). This provides the LLM with richer, more relevant context than standard chat histories.

Crucially, the state machine is defined as data (e.g., in JSON), allowing engineers to swap out the underlying LLM or the process logic independently.

Experimental Evaluation

The researchers evaluated SHERPA across three distinct domains to test its versatility:

  • Code Generation: Using the HumanEval benchmark (164 Python problems).
  • Class Name Generation: Using a Modeling Dataset of 135 classes across eight domains to test system design capabilities.
  • Question Answering: Using the Clevr dataset, which requires multi-step spatial and logical reasoning based on scene graphs.

Key Results and Discussion

SHERPA demonstrated superior performance compared to direct prompting in 12 out of 15 tested scenarios across different model families, including GPT-4o and Qwen-2.5.

Comparative performance of SHERPA vs. direct prompting on HumanEval, Modeling, and Clevr benchmarks.

Performance Gains

In code generation, SHERPA achieved an average improvement of 3.25 percentage points on HumanEval. The most significant gains were observed in tasks with established human best practices but limited LLM training data, such as class name generation, where SHERPA achieved a 6.58 point F1-score improvement. Even task-specific models like Qwen-2.5-Coder saw a boost, reaching a 91.26% Pass@1 on HumanEval when integrated into the framework.

Efficiency and Robustness

SHERPA acts as a "force multiplier" for smaller models (e.g., GPT-4o Mini or Qwen-7B). While these models possess raw capability, they often lack the discipline to follow complex workflows; the state machine provides the necessary guardrails. Furthermore, by optimizing state machine configurations—such as a test-driven approach that avoids redundant generations—the authors reduced LLM API calls by up to 50% without sacrificing quality.

Conclusion and Future Work

SHERPA demonstrates that model-driven engineering principles can significantly enhance the reliability and efficiency of LLM-based agents. While the framework currently relies on manually designed state machines, future work aims to automate the generation and optimization of these models using LLMs themselves. Additionally, the authors plan to integrate user-friendly modeling languages like PlantUML and explore Reinforcement Learning (RL) to further refine policy navigation within the state machines.

Summary updated

BibTeX
@inproceedings{chen2025sherpa,
  author = {Boqi Chen and Kua Chen and José Antonio Hernández López and Gunter Mussbacher and Dániel Varró and Amir Feizpour},
  title = {{SHERPA: A Model-Driven Framework for Large Language Model Execution}},
  booktitle = {Models},
  year = {2025},
  doi = {10.1109/models67397.2025.00024},
}
Figure