Before Blaming the LLM, Diagnose the Workflow: A Role-Level Difficulty Profile for Translating Natural Language into Ontology-Grounded Statements

Liang Zhang, Benoît Daireaux, Sergey AlyaevarXiv 2026 · arXiv

TL;DR

When an LLM fails to translate natural language into complex domain-specific statements, we usually blame the model's "intelligence." This paper argues that the fault often lies in the workflow structure itself and provides a diagnostic "difficulty profile" to help engineers identify whether they need a better prompt, a cleaner ontology, or a human-in-the-loop.

What Problem Does This Paper Address?

Translating natural language into structured, ontology-grounded statements—specifically Assertion Box (ABox) generation—is a high-stakes task in fields like engineering and medicine. Unlike simple Text-to-SQL, where success is often defined by whether a query executes, ABox generation involves mapping informal descriptions to a pre-existing, rigid vocabulary of concepts and relations.

The authors point out a recurring frustration: an LLM might produce a statement that is syntactically perfect (it follows the grammar) but semantically useless (it uses the wrong concepts or misses implicit domain requirements). Currently, practitioners often treat these failures as "weak model performance." This paper argues that these failures are frequently caused by structural friction—like ambiguous inputs or gaps in the ontology—that no amount of model scaling will fix.

Methods

The authors propose a role-level difficulty profile. Instead of looking at the system as a single "black box," they break the workflow down into five recurring roles:

  1. Input: The natural language intent.
  2. Generator: The component (usually an LLM) that grounds input to symbols.
  3. Output: The resulting DSL/ontology artifact.
  4. Validator: The check for whether the output is "correct."
  5. Repairer: The mechanism for fixing failures.
Fig. 1. Typical workflow and recurring roles in LLM-based ontology-grounded semantic parsing.

To diagnose these roles, they introduce seven variables (e.g., input sufficiency, grounding complexity, template fit). For each variable, the framework provides "high-friction evidence" to look for and "redesign levers" to pull. For example, if the problem is Input Sufficiency, the lever isn't a better LLM; it’s an "intent gate" that asks the user for clarification before the model even tries to generate code.

Datasets

The paper does not introduce a new large-scale benchmark. Instead, it uses a real-world case study from the drilling and wells domain.

  • Ontology: A snapshot of the DDHub semantic interoperability framework (approx. 2,250 classes and 200 properties).
  • Task: Generating configuration assertions for drilling signals based on the D-WIS standard.

The authors use a specific "diagnostic case"—retrieving a "Bottom Hole Depth" signal—to show how a short, vague user request requires a surprisingly complex multi-step ontology mapping that is prone to failure without proper workflow support.

Key Insights and Ideas

  • Syntactic Validity $\neq$ Semantic Adequacy: In ontology-grounded tasks, the model can easily produce "valid" code that is "wrong" because it misses implicit domain assumptions (e.g., forgetting that a depth signal must be attached to a specific reference frame).
  • Structural Friction: Difficulty is often a property of the task setup, not just the model. If the ontology has many overlapping or "near-match" terms, the Grounding Complexity is high, and the model will likely struggle regardless of its size.
  • The "Template Fit" Problem: If a domain hasn't stabilized its modeling patterns into clear templates, the LLM is forced to invent ad hoc structures. The authors suggest building a "motif library" to guide the model toward consistent, expert-approved patterns.
Fig. 1. Typical workflow and recurring roles in LLM-based ontology-grounded semantic parsing.

Key Results

The authors demonstrate the profile's utility by analyzing a failed generation attempt in the drilling domain.

  • The Failure: The LLM produced a simple signal-to-quantity mapping.
  • The Diagnosis: Using the profile, they identified that the input was underspecified (missing reference frames) and the template fit was weak.
  • The Solution: Rather than "blaming the LLM," the profile suggested specific redesigns: adding an intent gate to catch underspecified inputs and creating a "repair trace" so a human expert can see why the model chose a specific (incorrect) path.

Limitations and Future Work

The authors acknowledge that this is a qualitative framework, not a quantitative metric. It relies on the judgment of practitioners and does not yet "predict" model accuracy in a mathematical sense.

Future work involves:

  • Conducting quantitative studies to see if different engineers arrive at the same difficulty profile for the same task.
  • Testing how well the profile transfers to other complex domains like healthcare or legal tech.
  • Integrating the profile into automated "human-in-the-loop" systems that can trigger expert review only when specific high-friction variables are detected.

Summary updated

Abstract

Natural-language-to-domain-specific-language (NL-to-DSL) semantic parsing systems increasingly translate informal user descriptions into structured artifacts such as Structured Query Language (SQL) queries, Application Programming Interface (API) calls, workflows, formal specifications, and ontology-grounded domain-specific language (DSL) statements. This paper focuses on ontology-grounded Assertion Box (ABox) DSL generation: the translation of natural language domain descriptions into assertion-level DSL statements grounded in a pre-existing ontology vocabulary. In this setting, syntactic validity is often easier to achieve than semantic adequacy. Failures are frequently attributed to weak generator models, but recurring sources of difficulty are also structural: insufficient input, weak domain grounding, output template and vocabulary gaps, weak validation, and non-local repair. We propose a role-level difficulty profile for diagnosing difficulty in ontology-grounded NL-to-DSL semantic parsing at the workflow level. The profile organizes diagnostic variables across five recurring workflow roles: Input, Generator, Output, Validator, and Repairer. The seven variables are input sufficiency, context availability, grounding complexity, template fit, vocabulary coverage, validation strength, and error observability. Practitioners construct the difficulty profile through qualitative judgment over available evidence. This experience-informed, non-exhaustive difficulty profile is intended to support workflow diagnosis and guide redesign or human-in-the-loop intervention.
BibTeX
@misc{zhang2026before,
  author = {Liang Zhang and Benoît Daireaux and Sergey Alyaev},
  title = {{Before Blaming the LLM, Diagnose the Workflow: A Role-Level Difficulty Profile for Translating Natural Language into Ontology-Grounded Statements}},
  year = {2026},
  eprint = {2610.35267},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35267},
}
Figure