Learning When a Summary Token Is Grounded, From Incomplete Evidence

Pointer-generator summarizers let a model either copy a word straight from the source document or generate a word from its own vocabulary. They are trained on a quiet assumption: a summary word counts as "from the source" only if it appears there verbatim. Every other word is treated as something the model must produce from memory. In clinical dialogue summarization that assumption fails. A patient says "my tummy has hurt for two weeks," and the note says "abdominal pain × 2 wks." Every fact in that note comes from the source, yet almost no token matches. Standard training therefore pushes exactly these fact-bearing, reworded tokens into the free-generation path, where compact models invent, swap, or drop clinical details.

We propose to treat "is this summary token grounded in the source?" as a positive-unlabeled (PU) learning problem, meaning a classification problem where some examples are confirmed positives and the rest are an unlabeled mix of positives and negatives. Verbatim and dictionary-matched copies are the confirmed positives. Every other summary token is unlabeled: some are reworded source facts, some are connective phrasing, and some are unsupported content already present in the references. A grounding gate trained with a PU risk estimator learns to recognize reworded source facts. At decoding time it sends them to a source-restricted output path instead of the open vocabulary. The work tests whether this closes the factual gap between compact domain-tuned summarizers and larger general models on clinical dialogue.

Core Claim

The claim is narrow and falsifiable. When a copy gate is supervised by exact or dictionary matches and everything else is counted as "generate," the gate systematically under-predicts grounding for the tokens that matter most clinically: colloquial-to-formal rewordings, reformatted numbers, and inflected terms. Correcting that bias with a non-negative PU risk, rather than a uniform copy penalty or a training-free heuristic, should lower the rate of unsupported clinical concepts in the output of a sub-3B summarizer. It should do so at matched fluency, and the gains should concentrate on reworded facts rather than coming from blanket extraction.

Where This Sits

Three lines of work supply the mechanism, and each stops short of this question. Dr. Summarize adapts a pointer-generator to telemedicine dialogues. It adds a fixed penalty on the generation probability, a negation switch, and UMLS concept injection, and doctors preferred its output 37.1% vs 18.5% over a baseline. Its penalty is uniform, and its grounding signal comes only from concepts the ontology already lists. We replace the uniform penalty with a learned, prior-calibrated gate that can mark unlisted terms as grounded. Improving Latent Alignment in Text Summarization by Generalizing the Pointer Generator (GPG) diagnoses the same rigidity we target: verbatim copying cannot handle inflection or synonymy. It fixes this with latent relation vectors over aligned source words, reporting XSum ROUGE-1 of 33.11 against 31.87 and DUC 2004 alignment precision of 0.533 against 0.435. GPG changes what a copy can produce but still learns alignment only through likelihood. We change which tokens the gate is taught to treat as grounded, and the two ideas could be combined. LargePiG shows that pretrained LLMs already contain a latent pointer: a copy probability can be read from the divergence between intermediate and final layers and applied through attention, without training. It gives no way to inject a domain dictionary or adjust to a domain's grounding rate, so it serves as our training-free competitor.

The PU side supplies the estimator and also sets limits on the design. Analysis of Learning from Positive and Unlabeled Data shows that treating unlabeled data as negative with convex losses biases the decision boundary, to the point of total failure at high class prior. That is our core motivation: exact-match copy supervision is exactly this "unlabeled as negative" setup. Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning (AdaPU) applies a bounded non-negative PU loss to dictionary-matched entities and reaches CoNLL PER F1 of 90.17 against 95.71 fully supervised. It works only on tagging and reports trouble with multi-word boundaries. We reuse its bounded loss but apply it to decoder states, which already carry sequential context. PU-Bench constrains us in two ways. It finds that plain nnPU stays competitive with newer methods, so we use it rather than an elaborate variant. It also finds that PU estimators degrade when labeling is not completely at random. Our labels certainly are not random, since numbers and names get copied verbatim far more often than symptoms, so we treat this as a primary risk. A Closer Look to Positive-Unlabeled Learning from Fine-grained Perspectives shapes the training recipe: moving-average targets and hard pseudo-labels hurt in PU settings, so we avoid both.

The domain-specialization evidence sets the application regime. PULSAR at MEDIQA-Sum 2023 found that general Flan-T5-11B beat its MIMIC-pretrained counterpart on Task B, ROUGE-1 50.75 vs 49.20, which the authors attribute to a mismatch between inpatient notes and outpatient dialogue. HuskyScribe found that fine-tuned T5-large was competitive with prompted GPT-3.5 but still hallucinated in Assessment and Plan sections. Both results point to register mismatch, colloquial speech against formal notes, as the unsolved issue. That mismatch is precisely what makes verbatim copy supervision incomplete.

Research Program

The model is a compact encoder-decoder (Flan-T5-large and Flan-T5-3B) fine-tuned for dialogue-to-note summarization. We add a light pointer layer, following the small copy layer of A Copy-Augmented Generative Model for Open-Domain Question Answering, and a grounding gate $g_t \in [0,1]$ computed from the decoder state at step $t$. The output distribution is

$$P(w) = g_t \, P_{\text{src}}(w) + (1-g_t)\, P_{\text{vocab}}(w).$$

Here $P_{\text{vocab}}$ is the ordinary vocabulary distribution. $P_{\text{src}}$ mixes two things: copy attention over source tokens, and the vocabulary distribution renormalized over a candidate set $C(x)$. The set $C(x)$ contains source tokens, their lemmas and number normalizations, and the formal synonyms UMLS lists for any source mention. The source-restricted path can therefore produce "abdominal" from "tummy" whenever the dictionary links them. It can never produce a concept absent from both the source and its dictionary neighborhood.

The gate is trained with an auxiliary loss added to the usual likelihood. Positives $\mathcal{P}$ are target tokens that are verbatim, lemma, number, or UMLS-synonym matches of some source span. Unlabeled tokens $\mathcal{U}$ are all remaining target tokens. With $\pi$ the fraction of unlabeled tokens that are truly grounded and $\ell$ the bounded mean-absolute-error loss used by AdaPU, the gate loss is the non-negative PU risk

$$R(g) = \pi\,\mathbb{E}_{\mathcal{P}}[\ell(g,1)] + \max\!\big(0,\ \mathbb{E}_{\mathcal{U}}[\ell(g,0)] - \pi\,\mathbb{E}_{\mathcal{P}}[\ell(g,0)]\big).$$

The clipping at zero prevents the overfitting to negatives that unbiased PU risks suffer in neural networks.

Because positives are not selected at random, we do not use a single $\pi$. We bucket tokens by coarse type (number, UMLS-typed concept, other content word, function word) and estimate one prior per bucket. Two estimates are compared: tree-induction prior estimation, TIcE, run on gate features, and direct counts from a small hand-annotated set of about 300 reference notes. In these notes annotators mark each target token as grounded-and-reworded, free phrasing, or unsupported. The same annotated set is the ground truth for evaluating the gate itself. We train with soft targets only and no moving-average teacher, as the fine-grained PU study recommends.

We also examine one side effect. Tokens the PU gate scores as confidently not grounded, yet which carry clinical concepts, are candidates for unsupported reference content. We report how often that holds. We do not use it for training in the main system, so the result stays uncontaminated.

Baselines

The naive reference point is the same backbone fine-tuned with ordinary cross-entropy and no pointer layer. It tells us how much hallucination exists before any grounding mechanism is added.

The most direct ancestor is the Dr. Summarize recipe: a pointer layer with a uniform penalty on the generation probability plus UMLS concept boosting, reimplemented on our backbone. It is the strongest competitor because it already attacks clinical hallucination through the copy gate. Beating it with a learned gate is the heart of the claim. We tune its penalty weight on the same validation set, with the same number of trials as our prior and loss-weight search. Its reported result, 37.1% doctor preference against 18.5% for its baseline on the Curai telemedicine data, cannot be compared numerically with MEDIQA. We will use Curai only if we can obtain access.

The strongest reported number on our main benchmark is PULSAR's general Flan-T5-11B at ROUGE-1 50.75 on MEDIQA-Sum 2023 Task B. We rerun Flan-T5-11B fine-tuned under our protocol. Beating it means our 3B or smaller model matches it on ROUGE-1 within one point while having a clearly lower unsupported-concept rate. We do not expect to exceed it on ROUGE. We run LargePiG training-free on an instruction-tuned 7B model with few-shot prompts, as both a prompted-LLM reference and a training-free pointer reference. The file has no reported result on Lacuna for LargePiG on clinical summarization, so we produce that number ourselves. Where compute allows, GPG serves as the rigidity-fixing competitor on XSum. We compare against its reported ROUGE-1 of 33.11 and also rerun it on our backbone.

The ablation that isolates the key ingredient keeps everything identical, including the positive set, $C(x)$, and the restricted path, and swaps the PU risk for plain cross-entropy that treats all unlabeled tokens as negative. A second ablation uses the PU gate but gives the grounded path the full vocabulary. This separates the effect of better routing from the effect of restricting outputs.

Why not just…?

Why not just expand the matching heuristics and treat the result as complete labels? This is the most serious threat. Our positive set already includes lemma, number, and UMLS-synonym matches, so the PU correction acts only on what those heuristics miss. Our bet is that the residual (colloquial descriptions, patient idioms, drug nicknames, relative dates) is exactly where the clinical errors live, because it is the part not covered by UMLS. The naive-negative ablation settles the question directly. If it matches the PU gate on gate AUC over annotated reworded tokens and on unsupported-concept rate, the PU framing adds nothing, and we will report that dictionary expansion suffices.

Why not just penalize generation uniformly, as Dr. Summarize does? A uniform penalty cannot tell connective phrasing from reworded facts. It raises copying everywhere, which we expect to show up as a lower share of novel n-grams and choppy notes: the extractive bias pointer models are known for. We measure the novel n-gram share alongside factual metrics. The PU gate must lower unsupported concepts at an abstractiveness level equal to or higher than the penalty's. Matching it only at the cost of more extraction would count as a loss.

Why not just use a training-free pointer like LargePiG on a bigger model? It needs no training and was shown to beat DoLa on truthfulness. However, its copy signal comes from the model's own attention, and it cannot use a domain dictionary or a domain's grounding rate. The reworded-fact case requires knowing that "tummy" licenses "abdominal," and nothing inside the model supplies that link reliably. If LargePiG on a 7B model matches our 3B system on unsupported-concept rate, the contribution shrinks to efficiency, and we will say so.

Why not just prompt a large general model? HuskyScribe found fine-tuned T5-large competitive with GPT-3.5, and PULSAR found scale alone did not close the gap on data-poor Task C (ROUGE-1 27.64 before augmentation). Those results concern ROUGE, not faithfulness, so the prompted 7B baseline stays in the comparison.

Proposed Experiments

The primary benchmark is MEDIQA-Sum 2023 Task B, with Task C as a data-poor stress test. We report ROUGE-1 for comparability, plus three faithfulness measures. The first is the unsupported-concept rate: the share of UMLS concepts in the summary with no source mention or dictionary neighbor, audited by a clinician on a sample. The second is negation accuracy on annotated findings. The third is a blinded clinician count of fabricated or swapped facts on 100 notes per system. XSum serves as an out-of-domain check, since its references are known to contain unsupported content that falls in our unlabeled set.

Four outcomes are decisive, and each could come out against us. First, on the annotated reference tokens, the PU gate must separate grounded-reworded from free tokens better than the naive-negative gate, measured by AUC, with the gap concentrated in the colloquial-reworded bucket. Second, the unsupported-concept rate must fall relative to both Dr. Summarize and the naive-negative ablation, at an equal or higher novel n-gram share. Third, a 3B model with the PU gate must come within one ROUGE-1 point of fine-tuned Flan-T5-11B while hallucinating less. That would reframe the PULSAR paradox: general scale buys ROUGE, while grounding buys faithfulness. Fourth, a prior-sensitivity sweep will scale each bucket's $\pi$ by 0.5 to 1.5 and compare TIcE against annotated priors. This shows whether, given the non-random selection PU-Bench warns about, the method holds up or collapses when $\pi$ is underestimated. If it collapses, the per-bucket prior is a load-bearing part of the method and must be treated as one.