When a language model simplifies a hard sentence, the risky step is usually not swapping a rare word for a common one. It is restructuring: breaking one long sentence into several short ones, turning passives into actives, or pulling a relative clause out on its own. Each of these moves forces the model to decide again which noun is the agent, which outcome belongs to which treatment arm, and whether "because" survives the split. We propose that a large share of simplification hallucinations are errors of exactly this kind, which we call role-binding errors: the right words appear, attached to the wrong participants or linked by the wrong relation. If so, they can be caught with a precise, structural check. Parse the source and the output into small "who-did-what-to-whom" records, compare the two sets of records, and send only the mismatched records back to the model for repair.
This matters because the field's current instruments miss these errors. SARI, the standard simplification score, correlates near zero or negatively with factuality (Evaluating Factuality in Text Simplification). The best learned detector in that same study reaches only 19.8 F1 on substitution errors, the category where swapped roles fall. Entailment-style metrics such as AlignScore track system-level factuality but fail on individual medical summaries (FactPICO). Meanwhile, LLMs still lag human editors at the very operations (voice change, relative-clause reduction) that move arguments around (Evaluating LLMs for Portuguese Sentence Simplification with Linguistic Insights). We first measure whether syntactic operations cause role-binding errors, separately from sentence length, and then test whether a structural check repairs them better than generic faithfulness checking given the same compute.
Core Claim
Holding token length and content fixed, the rate at which an LLM swaps semantic roles, detaches modifiers, or drops discourse relations during simplification rises with the source's syntactic complexity and with how much restructuring the model is asked to do. These errors are invariant violations. The source's predicate–argument structure, meaning the set of events with their participants and the relations linking events, should survive simplification even when every sentence boundary changes. A comparison over these records will therefore detect and localize them better than whole-sentence entailment. Repair targeted at the mismatched records will cut human-judged substitution and insertion errors without falling back on the conservative copying that makes extractive systems safe but useless. FactPICO's Alpaca is the example: it scored 3.53/4.0 on factuality while barely simplifying.
The closest work differs in where the structure is used. DEPTH prunes a sentence to the shortest dependency path between two entities before classifying their relation. That path is the chain of grammatical links connecting them. DEPTH cut SciERC hallucination rates from 96.9% to 7.1%, which motivates our claim that syntactic clutter drives role confusion. But DEPTH never produces readable text and never checks whether its own pruning distorts meaning. AMRS³ uses Abstract Meaning Representation graphs (a parse of a sentence into a graph of concepts and their roles) to generate splits. It is evaluated only with BERTScore and a syntactic-complexity analyzer, never for factual errors. We use the graph as a check on a free-form LLM rewrite, not as the generator. Discourse-Based Sentence Splitting shows that predicting the discourse structure before splitting preserves causal and temporal links better (D-ACC 0.64 vs. 0.55). We adopt its relation inventory as part of our records, but we verify after generation rather than constraining before it.
Research Program
The work has three parts. The first is a diagnostic set that separates syntax from length. We start from MinWikiSplit, whose minimal propositions (3.84 short sentences per complex source) give a gold list of facts for each item. For each item, annotators and a rule-based generator produce several complex renderings of the same propositions, verified by humans. The renderings match in token count and differ in construction: coordination, passive voice, center-embedded relative clauses, long-distance dependencies, and connective-linked clauses. We add a clinical slice from Evidence Inference abstracts, the FactPICO source. There we hand-encode PICO bindings (which population got which intervention, and which outcome is attributed to which arm) as records, since MEDIMP is cited for exactly this kind of donor-versus-patient confusion. Models are then prompted under forced operations: lexical-only, split-only, voice-change-only, and free simplification. This lets us estimate error rates per operation, which Devaraj et al. and FactPICO left entangled.
The second part is the record check and its metric. A semantic role labeler and an AMR parser each map a text to a set of records $R$. Each record is a predicate with its labeled arguments, with negation and modality marked; discourse relations between records are taken from the discourse-splitting inventory. For a source $s$ and output $o$, we align records by predicate and argument head and define the role-binding error rate as
$$\mathrm{RBER}(s,o)=\frac{|\mathrm{swap}(R_s,R_o)|+|\mathrm{drop}(R_s,R_o)|+|\mathrm{ins}(R_s,R_o)|}{|R_s|}.$$
Here swap counts aligned records whose arguments are reassigned or whose polarity or modality changed, drop counts source records with no match, and ins counts output records with no grounding in the source. Coreference is resolved before alignment, so that "she" in a new sentence binds to its antecedent. This matters because splitting is precisely where pronouns break (Bingel and Søgaard).
The third part is targeted repair. The LLM writes a simplification. The checker returns the specific violated records, for example "in the source, the control group, not the treatment group, had the reduced rate." The model then rewrites only the affected sentences. The loop stops after at most two rounds. Insertions are allowed only if marked as explanations, so that we can separately audit the elaborative additions FactPICO found erroneous 31.3% of the time for GPT-4.
The main risk is that parsers fail most on the complex sentences we care about, which would make the checker blind exactly where errors happen. We measure this directly. Parser agreement and accuracy against gold records are reported as a function of construction type. Records that the role labeler and the AMR parser disagree on are flagged as uncertain rather than as errors.
Baselines
The naive reference is a single-pass zero-shot simplification from the same LLMs (one open model at 7–8B and Qwen 2.5 32B, plus GPT-4o-mini) with an explicit instruction to stay faithful. It is the condition FactPICO and the Portuguese study evaluate, and it fixes the error floor that any extra machinery must beat.
The strongest competitor is generic faithfulness reranking. It samples $k$ simplifications and keeps the one with the best AlignScore, or the best score from an NLI-based checker, against the source. This is the method a practitioner would deploy first, and it costs the same as our loop. We match compute by allowing it the same number of LLM calls our method uses (generation plus repairs, typically three). FactPICO reports only that AlignScore correlates moderately at the system level and poorly at the instance level; no reported reranking result on simplification was found on Lacuna. We count it as beaten only if human-judged substitution-plus-insertion errors (Devaraj taxonomy, severity ≥1) fall significantly lower under our method at equal calls, while human simplicity ratings stay within noise.
Two structure-first systems are also run. One is AMRS³ as described in its paper. The other is the pipeline from discourse-based splitting, whose reported discourse accuracy is 0.64 D-ACC on synthetic data. Both are scored with our human protocol and with D-ACC on the 352-triple gold test set. For information loss we also report Edit-Constrained Decoding, which reports 4% information loss against 15% for prior systems on Turk. It addresses lexical edits, so we expect it to be safe but to split rarely. Its split rate is reported alongside its error rate so that safety from not restructuring is visible.
Two ablations isolate the key ingredient. An entity-only check keeps the record machinery but drops argument roles: it flags an output only when an entity disappears or appears. Repair driven by an untargeted signal tells the model "some fact is wrong, revise" without saying which record. If either ablation matches the full system, role binding is not doing the work. For detection alone, we compare RBER against the RoBERTa classifier of Devaraj et al. on their annotated data, where the reported number to beat is substitution F1 = 19.8.
Why not just…?
Why not just prompt a frontier model to be careful, with chain-of-thought? The evidence suggests prompting does not close this gap. GPT-4's added explanations were wrong 31.3% of the time in FactPICO. A single-agent GPT-4o succeeded on only 48% of conditional-logic decompositions in A Hybrid Multi-Agent Prompting Approach. Still, our naive baseline includes a faithfulness instruction and a chain-of-thought variant. If those match the repair loop on RBER and on human errors, the conclusion is that modern models do not need structural feedback, and we would report that.
Why not just rerank or filter with an NLI or AlignScore checker? This is the real threat. Entailment models may already penalize role swaps because swapped sentences often contradict the source. Our argument against it rests on instance-level weakness (FactPICO) and on the 19.8 substitution F1. Neither measures reranking directly, so the compute-matched comparison above is what settles the question. We also run the hybrid, entailment rerank followed by record repair. If our method only adds value combined with entailment, the claim narrows to "role records localize errors that entailment ranks but cannot point to."
Why not just split first and simplify vocabulary second, as two separate prompts? Staging addresses the lexical–syntactic decoupling seen in Controllable Sentence Simplification via Operation Classification. But it does not check the split itself, and the split is where our diagnostic predicts most role errors arise. We run staged prompting both with and without the record check after the split step. The difference between those two conditions measures the check's contribution independently of staging.
Proposed Experiments
The first decisive check is causal. On the matched-length diagnostic set, does RBER increase with construction complexity, and does it increase under split and voice-change operations relative to lexical-only, at the same token count? A flat curve would mean the "syntactic" hallucination story is really a length story. That would argue against structural repair, and we would redirect the work toward length control. No reported role-swap rate under forced operations was found on Lacuna, so this measurement is new.
The second check validates the metric. On 600 outputs drawn from ASSET sources (20.2% of whose references split), MinWikiSplit, and the clinical slice, two annotators label errors with the Devaraj taxonomy plus a role-swap tag. We report instance-level correlation and detection F1 of RBER against AlignScore and the RoBERTa classifier, broken down by construction type and by parser confidence.
The third check is end to end. On ASSET, the discourse-splitting gold set, and Evidence Inference abstracts in the FactPICO setup, we compare all baselines on human error counts, D-ACC, PICO-binding accuracy, and simplicity. Simplicity is measured by human rating and by L2SCA syntactic indices; SARI is reported only for continuity, given its known failure on ASSET (ASSET). The proposal succeeds if three things hold: targeted repair beats compute-matched entailment reranking on substitution-plus-insertion errors, the entity-only ablation recovers much less of that gain, and simplicity does not regress toward copying the input.
References
- Evaluating Factuality in Text Simplification
- FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence
- Evaluating LLMs for Portuguese Sentence Simplification with Linguistic Insights
- Hallucination-Resistant Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement
- Semantic Graphs for Syntactic Simplification: A Revisit from the Age of LLM
- Discourse-Based Sentence Splitting
- MinWikiSplit: A Sentence Splitting Corpus with Minimal Propositions
- MEDIMP: 3D Medical Images and clinical Prompts for renal transplant representation learning
- Text Simplification as Tree Labeling
- Edit-Constrained Decoding for Sentence Simplification
- A Hybrid Multi-Agent Prompting Approach for Simplifying Complex Sentences
- Controllable Sentence Simplification via Operation Classification
- ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformations