When a language model turns a document or an API response into a neat JSON record or table, its output parses perfectly. Some of the values inside it may still be wrong. One production study found that 11–13% of table cells were silently corrupted even when the model's built-in schema mode guaranteed valid output (Structured Hallucination in Tool-Using Agents). Human-AI studies separately find that non-experts judge machine output by how it looks, while professionals check it against what they know. In one study, lay annotators found model-extracted clinical evidence informative 57.25% of the time, compared with 9.36% for certified coders (Can Current Explainability Help Provide References in Clinical Notes). Nobody has measured how these two facts interact. Does a clean table hide model fabrications from the people asked to check it? Does expertise protect against this? And can the output format be redesigned so that expert attention lands only where the model actually invented something?
We propose to make every field in a structured output declare its provenance. A field is either a verbatim copy of a located source span, which a machine can check exactly, or a derived value with a named transformation and cited inputs, which only a person can judge. The interface then sends only the derived fields to the human verifier. The central claim has two parts. Standard structured presentation lowers laypeople's rate of catching errors relative to prose while leaving experts' rate roughly unchanged. Splitting fields by provenance raises the rate at which experts catch errors and cuts their time per record. Adding explanations does neither: it makes experts slower without making them more accurate, as the fraud-analyst study found (On the Importance of Application-Grounded Experimental Design).
Core Claim
The three research directions play different roles here. Hallucination inside structured fields is the failure we target. Structured output adherence, meaning methods that force a model to emit output matching a fixed format, is the mechanism we modify: the grammar now encodes where each value came from, not just the shape of the record. The difference between experts and laypeople is the setting in which the intervention is judged, because the value of a verification interface depends on who is verifying.
The specific new claim is this. The useful signal for human verification of structured extractions is not an explanation. It is a split between fields that can be checked mechanically and fields that require judgment. Once that split is exposed, the mechanical fields can be verified automatically and the expert's effort goes to the smaller judgment set.
Prior work motivates this claim but does not test it. The synthesis-corruption paper measures cell corruption with no human in the loop, on one model and one domain. ExtractBench separates omitted fields from fabricated ones, but only with automated metrics. The xRAC study and Can GPT-4 Sway Experts' Investment Decisions? measure how experts and laypeople differ, but only on unstructured snippets and prose.
Research Program
The system has three parts.
Provenance-typed output grammar. We wrap any target schema so that each leaf value must be emitted in one of two forms. The first is a copy form holding a character span and the copied string. The second is a derive form holding a value, an operation from a small fixed vocabulary, and the spans it depends on. Operations include unit or date normalization, arithmetic, code lookup, and free inference.
Input-dependent decoding constraint. For open-weight models, grammar-constrained decoding restricts the model's next token to those allowed by a formal grammar. Following the input-dependent grammars of Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning, we build the copy branch from the source text itself, so a copy field can only contain an actual substring. That paper reports that restricting candidates this way raised entity disambiguation accuracy from 54.1% to 80.3%. We implement the grammar with Transformers-CFG.
Under this constraint, value mutations like those in the flight study are impossible in copy fields. Any change has to appear as a declared derivation. Where the declared operation is mechanical, such as a unit conversion or a date shift, we recompute it and flag any mismatch. Derived fields labelled as free inference go to the human, together with their cited spans.
Triage interface. The verifier sees the record with copy fields collapsed and marked as machine-verified. Derived fields are expanded, each next to its cited source spans.
The research is the controlled comparison of this interface against the obvious alternatives, run separately with experts and with laypeople. We also measure how much the provenance grammar itself costs in extraction accuracy. That cost is a real risk: ExtractBench reports that provider structured-output modes reduced accuracy, and StructTest reports that most open-source models lose about 70% accuracy when format rules move from easy to hard.
Baselines
Naive reference point. The naive reference is the provider-native schema mode rendered as a plain table. Verifiers are told only to check the record against the source. This is what practitioners ship today. Its automated corruption rate is known: 11–13% cell-level corruption for native response_schema on the frozen flight benchmark (Structured Hallucination). The same model, prompts, and source documents feed every arm.
Prose presentation. A prose rendering of the same extraction lets us test the masking hypothesis. The contrast between table and prose is what shows whether structure itself hides errors from laypeople.
Strongest existing assistance method. The strongest existing method is the table plus per-field evidence highlights, in the style of xRAC rationales and the professional-coder evidence spans in MDACE. This is the standard way to "ground" an extraction for a human. Its known expert result is negative: coders rated distilled-model rationales informative 9.36% of the time, against 57.25% for laypeople. For a human baseline on detecting errors in structured extractions, we found no reported result on Lacuna, so we measure it ourselves.
Post-hoc provenance. The ablation that isolates the decoding constraint keeps the triage interface identical. It computes the copy-versus-derived split after generation, by fuzzy-matching each value from native schema output against the source, instead of forcing the model to declare provenance while decoding. If this ablation matches the full system on expert detection and time, the constraint adds nothing, and we will recommend the cheaper post-hoc version.
Automated upper bound. In the flight domain, where the source is already structured, SSE Metadata Injection serves as an upper bound. This method streams the API data directly to the frontend and bypasses LLM generation. It reports 0% corruption and 3.0s against 11.4s latency.
Fairness and the bar to clear. All arms use the same model checkpoint, the same source documents, and the same seeded error set. Prompt tuning gets an equal budget: a fixed number of development documents per arm. Human arms are counterbalanced within subjects. Beating the strongest baseline means two things. First, the full system shows higher expert detection recall of corrupted fields than the evidence-highlight arm, with a false-alarm rate that is no higher and a median time per record that is lower. Second, the full system shows extraction accuracy within a pre-registered margin of native schema output.
Why not just…?
Why not just validate the JSON and retry on failure? The corrupted outputs in the flight study passed parsing every time, so a retry triggered by validation failure never fires. IterGen also documents that full restarts throw away valid work and often reproduce the same error. Validation checks shape. Our target is wrong values inside a valid shape.
Why not rely on the provider's native structured-output mode? It is our naive baseline, and published evidence already shows it fails in both ways that matter. It leaves 11–13% corrupted cells, and ExtractBench reports it lowering extraction accuracy on complex schemas. The fair question is whether our own grammar pays the same cost. We measure that directly on ExtractBench's credit-agreement and resume domains.
Why not just use retrieval-augmented generation? Synthesis corruption occurred even when the model had the exact correct API payload in its context, so retrieval does not remove it. Fine-tuned retrieval in Reducing hallucination in structured outputs via RAG still left up to 7.5% hallucinated steps. Retrieval supplies inputs to the model. It does not tell the human which outputs to check.
Why not string-match values against the source after generation? This is the real threat, and it is why post-hoc provenance is a formal ablation. Post-hoc matching cannot tell a legitimate normalization, such as a date reformatted or a currency converted, from a mutation. We expect it to push many correct derived values into the human's queue as false alarms, or to wave through near-matches as copies. The declared derive form makes the transformation explicit and recomputable. If the ablation performs as well anyway, the contribution shrinks to the interface finding, and we will say so.
Why not show rationales or evidence highlights for every field? Experts reject such rationales at high rates. In the fraud study, feature-attribution explanations added 14–16 seconds per decision without improving dollar regret, while showing only the model's score cut decision time from about 79 to 51 seconds. Our design follows the lesson of that score-only arm: show less, but show the right thing.
Why not run the study on crowdworkers? Lay and expert judgments overlapped with Jaccard similarity below 40% in the xRAC study. Amateurs were swayed by GPT-4 far more than veterans, whose persuasion rate was 15.6%. We include laypeople as a group precisely because we predict the intervention helps the two groups differently.
Why not bypass the LLM entirely, as SSE injection does? That works only when the source is already structured and the frontend has a fixed renderer for it. Clinical notes and credit agreements have no payload to inject. We keep SSE as the automated ceiling in the flight domain only.
Proposed Experiments
Automated study. We run open-weight models under the provenance grammar and under native-style schema output on three sources. The first is the frozen flight benchmark: 50 queries with payloads of 1 to 118 flights. The second is MDACE inpatient charts, where coder evidence spans give gold provenance for copied values. The third is ExtractBench's credit-agreement and resume domains. We report four measures:
- cell corruption in copy fields, which is zero by construction and is checked as a sanity test;
- corruption in derived fields, with factual consistency against cited spans scored using AlignScore;
- the fraction of fields that land in the copy form, since this determines how much the triage can save;
- field-level extraction accuracy against native output, to measure any formatting cost.
Human study. This is a within-subject design that crosses presentation arm (prose, plain table, table with highlights, post-hoc triage, provenance triage) with expertise group. The expert groups are certified medical coders for MDACE records and credit or legal analysts for the agreements. Laypeople see the same items. Each record contains the model's naturally occurring errors plus seeded value mutations at a known rate, so detection recall is measurable. The measures are:
- detection recall and false alarms per corrupted field;
- time per record;
- accuracy of the downstream decision, such as the correct final code or a correct covenant answer;
- post-task trust ratings, to capture the confidence-trap effect.
Decisive checks. The masking hypothesis passes only if laypeople's detection recall is lower on tables than on prose while experts show no comparable drop. The intervention passes only if provenance triage beats evidence highlights for experts on both recall and time, and beats post-hoc triage on false alarms. The project also has a pre-registered failure condition. If the provenance grammar cuts extraction accuracy by more than the gain in human-verified accuracy, the net effect is negative, and we will report that decoding-time provenance is not worth its cost.
References
- Structured Hallucination in Tool-Using Agents: Measuring and Mitigating LLM Synthesis Corruption in Production
- Can Current Explainability Help Provide References in Clinical Notes to Support Humans Annotate Medical Codes?
- On the Importance of Application-Grounded Experimental Design for Evaluating Explainable ML Methods
- ExtractBench: A Benchmark and Evaluation Methodology for Complex Structured Extraction
- Can GPT-4 Sway Experts' Investment Decisions?
- Grammar-Constrained Decoding for Structured NLP Tasks without Finetuning
- StructTest: Benchmarking LLMs' Reasoning through Compositional Structured Outputs
- MDACE: MIMIC Documents Annotated with Code Evidence
- IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking
- Reducing hallucination in structured outputs via Retrieval-Augmented Generation