Point to the Cue: Rule-Checked Assertion Status for Grounded Clinical Extraction

When a language model reads a radiology report and writes structured findings, it rarely invents the word "pneumonia". The more dangerous error is that it turns "cannot exclude pneumonia" into "pneumonia: present", or attaches "no" to the wrong finding. Copy mechanisms guarantee that the words come from the source, so they fix the first problem. They do nothing for the second. We propose a decoding scheme that targets the second. For every finding it extracts, the model must also copy the short cue phrase that justifies the finding's status, such as "cannot exclude", "no evidence of", or "consistent with". The status label is then assigned by a small rule table that reads only that cue. The model may write no status the rules cannot justify from a span it actually copied.

The underlying bet concerns where hand-written rules break. Rule labelers like CheXpert are brittle at localization and scope: they struggle to find which phrase in a messy sentence is the cue, and which finding it governs. They are much more reliable at interpreting an isolated cue phrase. So we let the language model do the locating, by pointing, and let the rules do the interpreting. A neural fallback handles cue phrases the rule table has never seen, and every fallback is logged as a candidate rule for human review. The result is extraction whose every status decision is tied to a verbatim source span and an auditable rule, run on an off-the-shelf open model without training.

Core Claim

In clinical report extraction, the remaining hallucinations after copying are mostly assertion hallucinations: wrong negation, wrong uncertainty, and epistemic overconfidence, where a hedge becomes a definite statement (Tang et al.). They also include wrong attachment between findings and modifiers, a form of the role swapping documented in MEDIMP.

We claim three things. First, forcing the model to emit a copied cue span before each status, and computing the status from that span with a rule table, cuts overconfident status errors well below free generation, quote-then-answer prompting, and attention-based pointing. Second, it does so without losing the recall that rule-only labelers lose when they move to a new hospital. Third, the cue phrases the model points to but the table does not cover form a short, high-yield list for rule maintenance.

The new ingredient is not copying, which is old. It is not grammar-constrained output either. It is the requirement that a categorical decision be licensed by a copied piece of evidence, interpreted by a rule rather than by the model's free choice.

Research Program

The output is a small grammar of records. Each record has a copied finding span, a copied anatomy span (optional), a copied cue span or the literal token NONE, and a status. Decoding uses a field-typed version of the pointer-generator mixture from Get To The Point. In that original design, a learned gate blends "generate from the vocabulary" with "copy from the source". Here the gate is set by the field type of the current position, written $f_t$:

$$P(w \mid f_t) = \mathbb{1}[f_t = \text{scaffold}]\, P_{\text{grammar}}(w) + \mathbb{1}[f_t = \text{span}]\, P_{\text{copy}}(w),$$

The terms are defined as follows. $P_{\text{grammar}}$ is the model's vocabulary distribution restricted to tokens the record grammar allows next. $P_{\text{copy}}$ is the model's distribution restricted to tokens that continue some substring of the source report, enforced with a substring index (a suffix automaton) over the report. The status field is not sampled at all. If the copied cue is in the rule table, the status is the table's entry. If the cue is NONE, the status defaults to present. If the cue is unseen, a fallback asks the model for the status given only the cue and finding, and flags the record.

This design resolves the tension in Dr. Summarize, which penalized generation everywhere and paid for it in fluency. It also reuses the index-pointing idea of Don't Parse, Generate!, which generated structural tokens freely and pointed for arguments. Unlike that work, our version needs no training, and well-formedness is guaranteed by the grammar rather than reached 98% of the time.

The rule table is seeded from the negation and uncertainty trigger lists of the CheXpert labeler (CheXpert). It is deliberately applied only to isolated cue strings, never to whole sentences. We will run this on Llama3-8B, the model Automated Spinal MRI Labelling found competitive with GPT-4 on report labeling. We will also test a LoRA-tuned variant to see whether light tuning on RadGraph training records improves cue localization.

Related work plays three roles. LargePiG (LargePiG) supplies the evidence that attention over the input is more faithful than the model's stored knowledge. It also constrains our design: its copy decision comes from layer-wise divergence and unguided attention, so it points faithfully to whatever span attention lands on, relevant or not. We therefore replace soft pointing with hard, field-typed pointing.

The tree-constrained pointer generator (TCPGen) motivates a different choice. It constrains copying to a fixed entity dictionary and reports a recall-versus-precision trade-off as copy strength increases. Our copy space is the report itself, so there is no dictionary to go stale and no strength knob to tune.

Two papers motivate the division of labor between model and rules. CheXpert++ found that clinicians preferred a neural imitation over the rules 60% of the time when the two disagreed. This warns that routing status through rules could reintroduce rule errors. Our hypothesis is that those errors are mostly scope errors that pointing removes, and we test this directly. Sun et al. (Linguistic Variability and the Brittleness of Rule-Based Systems) found that pattern matching breaks because it cannot isolate the core anchor from surrounding noise, which supports splitting localization from interpretation.

Baselines

The do-nothing reference is the rule-based CheXpert labeler run unchanged. On its own evaluation it reports a micro-average F1 of 0.969 (CheXpert). We expect it to remain hard to beat on in-distribution chest reports. It is the reference for whether our system gives up precision. Beating it means matching its uncertain-class precision within one point while exceeding its recall on the cross-site split.

The strongest existing method for fine-grained extraction with uncertainty is the RadGraph PubMedBERT plus DYGIE++ tagger (RadGraph). It reports 0.94 micro F1 for entities and 0.82 for relations on MIMIC-CXR. Its schema already includes observation uncertainty, and the same paper documents substantial drops when it moves from MIMIC-CXR to CheXpert-sourced reports. No transfer number appears in our file, so we will measure it ourselves. We use the released model and the same RadGraph train/test splits. Beating it means higher F1 on the uncertain and absent classes on the cross-institution test, with entity F1 no more than two points lower in-distribution.

The strongest LLM competitor is quote-then-answer prompting. The same Llama3-8B is asked for JSON records that include a verbatim evidence quote and a status. It uses the same few-shot examples and decoding budget as our method. Quotes are checked for substring membership afterwards, and records with unmatched quotes are dropped. We run two versions. One uses free generation. The other uses our copy constraint on quote fields but lets the model choose the status. That second version is the key ablation: it isolates rule-assigned status from evidence copying.

We also run LargePiG decoding on the same model and prompts as the attention-pointing competitor. We run an ablation that keeps the rule table but removes the copy constraint, and another that removes the neural fallback. We also run the "Summary + Query" prompting pipeline of Automated Spinal MRI Labelling for the 14-label comparison. All LLM systems share one base model, one prompt set tuned on MIMIC-CXR development data with an equal number of trials, and greedy decoding.

Why not just…?

Why not just ask the model to quote its evidence and verify the quote afterwards? This is the real threat, and it is why the copy-constrained, model-chosen-status ablation exists. Post-hoc verification drops findings whose quotes fail, which costs recall. It also says nothing about whether the status matches the quote: a model can faithfully quote "cannot exclude" and still write "present". If the ablation with copied quotes and model-chosen status matches our full system on overconfident errors, the rule step adds nothing. In that case we would report the contribution as constrained quoting alone.

Why not just use LargePiG's training-free pointing? It decides when to copy from internal layer divergence and what to copy from raw attention. Pointer work such as Don't Copy That! shows that attention often lands on the wrong span. LargePiG was evaluated on query generation and QA, not on assertion status. We expect it to copy finding names correctly and still mislabel hedges, and the per-class status results will show whether that holds.

Why not just fine-tune a tagger like RadGraph, or distill the rules as CheXpert++ did? These are strong in-distribution. CheXpert++ also shows that neural imitation of rules can be preferred over the rules themselves. But both output labels without a copied cue tied to each decision, and RadGraph is documented to degrade across institutions. The cross-site test is designed to settle whether training-free pointing transfers better. If RadGraph transfers equally well, our advantage reduces to auditability, and we will say so.

Why not just keep the rules? EdIE-R reaches 95.49–96.86% entity F1 on its home corpus (Text mining brain imaging reports), close to human agreement. However, the same work reports that the system needs manual recalibration when report templates change. The cross-site experiments measure exactly that recalibration cost.

Proposed Experiments

The first decisive check is assertion fidelity on RadGraph MIMIC-CXR test reports. We report F1 for each status class and an overconfidence rate: the fraction of radiologist-labeled uncertain or absent observations that a system outputs as present. Our system should cut the overconfidence rate relative to quote-then-answer with model-chosen status. If it does not, the core claim fails.

The second check is cross-institution transfer. We develop only on MIMIC-CXR and evaluate on the CheXpert-sourced RadGraph test set. In a harder setting, we apply the chest-derived rule table to the Edinburgh Stroke Study brain-imaging corpus. There we compare against EdIE-R's reported 92.61–96.39% label F1 and the 94.02% human agreement on labels. The prediction is that the rule-only systems lose recall, the tagger loses F1, and our system loses less than both because localization never depended on site-specific templates.

The third check is a disagreement analysis that tests the CheXpert++ concern. We take every case where our system and the CheXpert labeler disagree and have a clinician classify each error as a scope error or a cue-interpretation error. Our design predicts that most rule errors are scope errors, and that the remaining errors are concentrated in fallback cases. If interpretation errors dominate, routing status through rules is the wrong choice and the fallback should become the default.

The fourth check is rule maintenance. We prune the seed table by half, run on the new site, have a clinician review the flagged unseen cues, add the approved ones, and rerun. We measure accuracy recovered per hour of review against the 8% gain from three hours of active learning reported for CheXpert++.

Finally, we report 14-label F1 on the CheXpert 500-study test set against the rule labeler's 0.969. This shows whether the record format costs anything on the coarse task practitioners already use.