When a model writes a chest X-ray report, some sentences come from the image and the patient record in front of it. Others come from what reports usually say. Current tools catch the second kind by comparing the generated report against a reference report written by a radiologist. At the moment a model is actually used, no such reference exists. This proposal asks whether unsupported sentences can be flagged from the text alone, with no vocabulary, using sentence shape. Deception research suggests they might. It showed that fabricated human text has a measurable grammatical signature, for example more subordinate clauses and more over-justifying verb phrases. That signature survives even after every word is replaced by its part-of-speech skeleton.
The concrete plan has four parts. We generate reports with several medical models while systematically removing their inputs: the image, the structured patient record, or both. We label each asserted finding as supported or not, using a structured graph of findings extracted from the reference report. We also mark findings that persist when the evidence is removed. We then test whether a word-blind syntactic detector separates supported from unsupported sentences when both sentences assert the same finding. If it does, we get a cheap, reference-free hallucination flag that a generator cannot satisfy simply by naming the right diseases. If it does not, we get a clean negative result showing that the human deception analogy does not carry over to language models.
Core Claim
The claim is that sentences a clinical report generator produces without support from its inputs differ in syntactic form from supported sentences, even with content held fixed. Content held fixed means the same finding, such as "small left pleural effusion," appears once as supported and once as hallucinated. A detector that sees only unlexicalized parse structure can tell the two apart. A second, independent signal comes from how plausible it is for the asserted findings to appear together. Combined with the syntactic signal, it should beat either one alone.
The psychological story behind human deception does not obviously apply to a language model. A model has no cognitive load and no motive to lie. Our mechanism is different and testable. When the model's evidence does not determine a sentence, it falls back on high-frequency report templates. We hypothesize that this leaves a structural trace, such as more formulaic phrasing, more categorical certainty, and less hedging, much as paper-mill manuscripts leave formulaic traces. The direction of each syntactic effect is an empirical question. The existence of a content-independent effect is the falsifiable claim.
Grounding in Prior Work
Three lines of work motivate the stylometric half. Finding Deceptive Opinion Spam by Any Stretch of the Imagination separated invented hotel reviews from real ones using psycholinguistic word categories plus word pairs, reaching 89.8% accuracy. Human judges reached only 53.1–61.9%. Syntactic Stylometry for Deception Detection went further. It showed that grammar rules read off parse trees reach 78.5% on the same data with all words stripped, and 91.2% when combined with unigrams. It also reported that syntax transferred across essay topics better than bag-of-words. Analyzing the Evolution of Scientific Misconduct Based on the Language of Retracted Papers found formulaic, low-diversity language in fabricated papers (F1 0.93). It also found that falsified data leaves little trace in prose, which is a warning we take seriously.
All three studies concern human authors with a motive to deceive. We transfer the measurement method, not the psychological theory.
The knowledge-graph half provides our labels, not our detector. RadGraph: Extracting Clinical Entities and Relations from Radiology Reports extracts a graph of anatomies and observations linked by typed relations, such as "located at," with relation-extraction micro F1 of 0.82. Its inter-radiologist relation agreement on CheXpert is only a Cohen's kappa of 0.397, which bounds how clean any label built from it can be. Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards used RadGraph overlap with the reference as a training reward. That raised factual correctness by 14.2% on MIMIC-CXR but produced list-like prose. Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation uses entailment-based factual scores in the same reference-dependent way. Every one of these requires the reference report. That is exactly what we remove at test time.
Two diagnostic papers supply the input-ablation design. The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness compared models given a full patient bundle against models given only a case ID. Ungrounded hallucination rates ranged from 5.9% to 91.5%, while grounded support rates compressed into 81.9–93.2%. That compression means grounded accuracy alone hides how much a model recycles its priors. HDI operates post hoc at the case level with an LLM judge. We move the same contrast to the sentence level and ask whether it can be predicted from text. MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks shows that strong models still favor fluent plausibility over grounding, with hallucination rates of 29.1% for Claude-4-Sonnet and 59.0% for Llama-4-Scout. It offers rubrics, not a detector.
Label dependency modeling enters as the second signal, and two papers constrain how we use it. FACT: Fuzzy Alignment with Comorbidity Topology for Reliable Multi-Label Medical Image Diagnosis showed that a disease co-occurrence graph improves image diagnosis and calibration, lowering calibration error on ODIR-5K from 27.51% to 16.22%. The same paper warns that noisy co-occurrence graphs degrade results. Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning warns that leaning on co-occurrence overfits. We therefore use the dependency model only as a conservative plausibility score on the set of asserted findings, never as a hard constraint.
Research Program
The first component is a corpus of generated reports with sentence-level support labels. We use the complete-modality cohort of When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion. It pairs MIMIC-CXR images with MIMIC-IV records across 7,149 stays, and we pair these with their reference reports.
Generators are chosen to span architectures and training styles. The first is a vision-plus-language-model report writer in the style of Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-ray Report Generation. The second is the Med-Flamingo setup with serialized patient-record prompts from MedPromptX: Grounded Multimodal Prompting for Chest X-ray Diagnosis. The third is a RadGraph-reward-trained model following Delbrouck et al. Each is run under four input conditions: full, image removed, patient record removed, and both removed.
A finding asserted in the full-input report receives two labels. It is unsupported if its RadGraph entity and relation have no match in the reference report's graph. It is input-insensitive if the model asserts it about as often without the evidence as with it, that is, if
$$\hat p(f \mid \text{ablated}) \ge \hat p(f \mid \text{full}) - \delta,$$
where $\hat p(f \mid \cdot)$ is the rate at which finding $f$ is asserted for that case across sampled generations, and $\delta$ is a small tolerance. The second label catches the prior recycling that HDI identified, including cases where the recycled finding happens to be correct. A radiologist-adjudicated subset of a few hundred sentences calibrates both automatic labels.
The second component is the detector. Each sentence is parsed, and we extract the unlexicalized and grandparent-augmented grammar rules of Feng et al. We add a small set of rhetorical features: hedging versus categorical certainty, negation, and clause depth. A linear model and a small tree-based model are trained on these features alone, so any signal they find cannot come from vocabulary.
The second feature family is a comorbidity-plausibility score. It is the log-likelihood of the asserted finding set under a sparse dependency model fit to CheXpert-style labels of the training reports. We regularize toward independence to respect the overfitting warnings above.
The third component is analysis. Which syntactic rules carry the signal? Does it survive when the generator was trained to write list-like text? Does it differ between image-unsupported and record-unsupported findings?
Baselines
The naive reference point flags a finding as hallucinated with probability equal to that finding's base hallucination rate in the training set. This captures the trivial fact that some findings, such as effusion or atelectasis, are hallucinated more often than others. Any detector must beat this per-finding prior. Otherwise it has only learned which words are risky. We report AUROC and AUPRC of every method against this prior, both overall and within each finding.
The strongest existing reference-free competitor is a clinical text encoder fine-tuned on the same sentence-level labels, with full access to vocabulary. No paper in our sources reports a sentence-level hallucination-detection result for radiology reports. The closest number is the 0.93 F1 for fabricated-paper detection reported in the scientific-misconduct study, which is a different task. The encoder gets the same training split, the same hyperparameter search budget measured in GPU-hours, and the same early-stopping rule as our detector.
A second strong competitor is an LLM judge given the image and the patient record and asked whether each sentence is supported. This is the adjudication style used in HDI and MHB. It has privileged access to the evidence, so it is a target to approach, not a fair peer. We report it alongside its inference cost.
The reference-based RadGraph matcher of Delbrouck et al. serves as an oracle ceiling, since it sees the reference report. Its reported downstream effect is the 14.2% factENTNLI gain on MIMIC-CXR. That is a generation result, not a detection number, so we report its detection accuracy ourselves.
The ablation that isolates the key ingredient is the content-matched comparison. We restrict evaluation to pairs of sentences asserting the same RadGraph finding, one supported and one not, and compare the syntax-only detector against the per-finding prior. Beating the encoder overall is not the claim, because the encoder sees content. The claim is that the syntax-only detector beats chance inside matched pairs, where the encoder's lexical advantage is neutralized. When the syntax features are added to the encoder, they should also give it a significant gain.
Why not just…?
Why not just fine-tune a clinical encoder on hallucination labels? This is the real threat. If the encoder's advantage comes entirely from knowing which findings tend to be fabricated, it will look strong overall and collapse inside content-matched pairs. Our matched-pair evaluation is designed to show exactly that. If instead the encoder also wins inside matched pairs and syntax adds nothing on top, then syntax is subsumed by learned representations. We would report that the stylometric hypothesis offers no practical value, even if the signal exists. The cross-topic transfer result in Feng et al. gives a reason to expect syntax to transfer better across generators and hospitals. We test this by training on one generator and evaluating on another, and by training on MIMIC-CXR and evaluating on Open-i.
Why not just ask an LLM judge with access to the image and record? Judges see the evidence, but they share the failure mode under study. CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays reports that Gemini-2.5-Pro completed its structured diagnostic path in only 17% of cases, with visual grounding the main bottleneck. MHB shows that judges themselves are drawn toward fluent plausibility. We measure judge accuracy directly. If a judge clearly dominates at acceptable cost, the syntactic detector's remaining value is as a fast first-pass filter, and we report it as only that.
Why not just use the comorbidity-implausibility score, or RadGraph against the reference? Reference matching is unavailable at deployment, which is the whole problem. Comorbidity implausibility only catches findings that clash with each other. A plausible but unsupported effusion passes it. FACT's warning about noisy graphs suggests it will be brittle across institutions. We report it alone, the syntax score alone, and their combination, so complementarity is measured rather than assumed.
Proposed Experiments
The decisive check is the content-matched evaluation on the CareBench complete cohort, pooled across the three generators. The syntax-only detector's AUROC within matched pairs must exceed 0.5 with a bootstrap confidence interval that excludes chance. It must also exceed the per-finding prior. If it does not, the core claim is false and we stop the stylometric line.
The second check separates the two kinds of support label. Is the signal stronger for input-insensitive findings, where the model is recycling priors, than for findings that are merely contradicted by the reference? A stronger signal for recycled findings would support the template-fallback mechanism over a general error mechanism.
The third check concerns the RadGraph-reward generator. If reward training pushes all sentences toward a uniform list style, the syntactic signal should weaken there. That would be an informative interaction between factuality training and detectability.
The fourth check is transfer. We train on one generator and test on the others, and we train on MIMIC-CXR and test on Open-i. Each transfer setting is compared against the fine-tuned encoder.
The fifth check is fairness and calibration. The misconduct study raised a concern about stylometry penalizing non-native writing. Our text is machine-generated, so the analogous risk is penalizing particular report-writing conventions. We test whether flag rates differ across the demographic subgroups that CareBench found fusion models already treat unequally, and we report calibration error for every detector.
The final check is a pilot on MedPromptX-VQA-style prompts drawn from HAIM-MIMIC-MM. It asks whether the same signal appears when a model explains a diagnosis rather than writing a report.
References
- Finding Deceptive Opinion Spam by Any Stretch of the Imagination
- Syntactic Stylometry for Deception Detection
- Analyzing the Evolution of Scientific Misconduct Based on the Language of Retracted Papers
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards
- Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation
- The Hallucination Dependence Index: A Cross-Condition Diagnostic for Clinical-LLM Faithfulness
- MHB: Medical Hallucination Benchmark for Large Language Models in Complex Clinical Tasks
- FACT: Fuzzy Alignment with Comorbidity Topology for Reliable Multi-Label Medical Image Diagnosis
- Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning
- When Does Multimodal Learning Help in Healthcare? A Benchmark on EHR and Chest X-Ray Fusion
- Activating Associative Disease-Aware Vision Token Memory for LLM-Based X-Ray Report Generation
- MedPromptX: Grounded Multimodal Prompting for Chest X-Ray Diagnosis
- CXReasonBench: A Benchmark for Evaluating Structured Diagnostic Reasoning in Chest X-rays