SHERPA: A Model-Driven Framework for Large Language Model Execution

Boqi Chen, Kua Chene, José Antonio Hernández López, Gunter Mussbacher, Dániel Varró, Amir Feizpour

TL;DR

SHERPA is a new model-driven framework that structures Large Language Model (LLM) execution using hierarchical state machines. By explicitly embedding human best practices and domain-specific workflows into these state machines, SHERPA significantly improves LLM performance on complex tasks like code generation and system modeling while providing a mechanism to reduce computational costs.

What Problem Does This Paper Address?

While LLMs are incredibly capable, they are fundamentally stochastic—they predict the next token based on probability, which often leads to "hallucinations" or a lack of long-term consistency in complex tasks.

The authors identify a major gap: structured reasoning. Standard prompting techniques like Chain-of-Thought (CoT) help, but they don't provide a rigorous way to enforce domain-specific "best practices" (e.g., the specific steps a software engineer takes to validate code). Furthermore, existing "agentic" frameworks often hard-code workflows, making it difficult for researchers to experiment with different logic structures without rewriting the entire application.

Methods

SHERPA (Stateful Hierarchical Execution and Reasoning for Process Automation) bridges the gap between Model-Driven Engineering (MDE) and LLM execution. Its architecture consists of three core components:

  1. Hierarchical State Machine (SM): Instead of a linear prompt, the task is decomposed into states (simple or composite). Transitions between states are triggered by events (like a user question or an LLM output). This allows for complex, nested logic—for example, an "Inspection" state that can loop back to a "Generation" state if errors are found.
  2. Policy Engine: This decides the "next move." SHERPA supports a hybrid policy: Rule-based (deterministic logic) and LLM-based (using the model to choose the best transition based on the current state and history). It also features a "fast-forward" mechanism to skip expensive LLM calls when only one transition is valid.
  3. Belief System: A structured memory that tracks the "trajectory" (the path taken through the state machine) and an "execution log" (inputs/outputs of every action). This provides the LLM with much better context than a simple chat history.

Crucially, SHERPA treats the state machine as data (defined in JSON), decoupling the workflow logic from the underlying code implementation.

Datasets

The researchers evaluated SHERPA across three distinct domains:

  • Code Generation: Using the HumanEval benchmark (164 Python programming problems).
  • Class Name Generation: Using a Modeling Dataset containing 135 classes across eight diverse domains to test system design capabilities.
  • Question Answering: Using the Clevr dataset, which requires multi-step spatial and logical reasoning based on scene graphs (e.g., "How many metal objects are there if you don't include spheres?").

Key Insights and Ideas

  • The "Wisdom" of the State Machine: The most significant gains occurred in tasks with established human best practices but limited LLM training data (like class name generation). By forcing the LLM to follow a "Noun Extraction -> Categorization -> Attribute Assignment" workflow, SHERPA achieved a 6.58 point F1-score improvement over direct prompting.
  • Smaller Models Benefit Most: One of the most striking findings is that SHERPA acts as a "force multiplier" for smaller, cheaper models (like GPT-4o Mini or Qwen-7B). These models have the raw capability but lack the "discipline" to stay on track; the state machine provides the necessary guardrails.
  • Efficiency by Design: By optimizing the state machine (e.g., a "test-driven" approach that avoids regenerating test cases unnecessarily), the authors reduced the number of LLM API calls by up to 50% without sacrificing output quality.
  • Separation of Concerns: Because the workflow is defined as a model (the SM), engineers can swap out the "brain" (the LLM) or the "process" (the SM) independently, enabling rapid experimentation.

Key Results

  • General Superiority: SHERPA outperformed direct prompting in 12 out of 15 tested scenarios across different model families (GPT-4o and Qwen-2.5).
  • Code Generation: Achieved an average improvement of 3.25 percentage points on HumanEval.
  • Cost Reduction: In the code generation task, the "test-driven" SM configuration achieved comparable performance to more complex "agent coder" approaches while requiring significantly fewer LLM invocations.
  • Robustness: Even when using task-specific models (like Qwen-2.5-Coder), adding the SHERPA framework further boosted performance, reaching a 91.26% Pass@1 on HumanEval.

Limitations and Future Work

  • SM Design Complexity: The framework's effectiveness depends heavily on the quality of the state machine design. A poorly designed SM can actually hinder high-end models (like GPT-4o) on tasks where they already excel.
  • Manual Modeling: Currently, the state machines must be designed by humans. The authors suggest future work could involve using LLMs to generate or optimize these state machines automatically.
  • Language Support: The current implementation uses JSON for SM definitions. The authors plan to integrate more user-friendly modeling languages like PlantUML or Umple to make the framework more accessible to non-experts.
  • Policy Evolution: They aim to explore Reinforcement Learning (RL) to train policies that can navigate the state machines even more effectively than standard LLM prompting.

Summary updated

BibTeX
@inproceedings{chen2025sherpa,
  author = {Boqi Chen and Kua Chene and José Antonio Hernández López and Gunter Mussbacher and Dániel Varró and Amir Feizpour},
  title = {{SHERPA: A Model-Driven Framework for Large Language Model Execution}},
  booktitle = {Models},
  year = {2025},
  doi = {10.1109/models67397.2025.00024},
}
Figure