Point to the Node, Then Say It: Graph-Constrained Copying for Faithful Generation from Underspecified Inputs

When a model writes text from structured data, such as knowledge-graph triples or an image's scene graph (a graph whose nodes are objects and attributes and whose edges are relations like "on" or "holding"), it faces two pressures that pull against each other. The input is deliberately underspecified. Nodes are bare concepts like eat-01, boy or Alan_Bean, with no tense, number, articles, or choice between "Alan Bean", "he" and "the astronaut". The generator must supply all of that. It must also add nothing that is not in the graph. Current systems satisfy the first pressure with language-model fluency and fail the second: open LLMs produce polished data-to-text output in which 76–86% of outputs contain a semantic error (Beyond Traditional Benchmarks). A pointer-generator (a decoder that at each step either writes a vocabulary word or copies an input token) is the classic fix for fidelity. But it copies surface strings, which fails when the source is a lemma that must be inflected.

We propose a decoder in which every content word must come from a pointing event aimed at a graph node, not at an input token. A separate realization step then turns that node into a surface form chosen from a set of licensed realizations: inflections of its lemma, its readable name, a pronoun, or a hypernym (a more general word, like "animal" for "dog"). Free vocabulary generation is restricted to function words and connective phrasing. The result has three properties. Entity hallucination becomes impossible by construction. Underspecification, meaning morphology plus the pragmatic choice between specific and vague reference, is handled explicitly and not left to language-model priors. Every mention carries a node identity, so faithfulness can be checked even when the text is intentionally vague.

Core Claim

The claim has two parts. First, separating which node is being talked about from how it is said lets a decoder guarantee entity-level faithfulness at small cost in fluency, with realization quality as the main thing to win. Second, the node trace this decoder emits supports a faithfulness evaluation that no longer penalizes accurate underspecified references. Existing hallucination scores and alignment models misjudge such references: CLIP prefers a wrong but specific caption over a correct vague one for 82% of images (Dealing with Semantic Underspecification in Multimodal NLP). The pointer is the core mechanism. Semantic underspecification defines what the realization step must solve. Scene-graph structure constrains where the pointer may go. Data-to-text faithfulness is the evaluation regime.

Relation to Prior Work

Several papers supply the method. The pointer-generator mixture of Get To The Point and its cheap cross-attention reuse in A Copy-Augmented Generative Model for Open-Domain QA show that copying is inexpensive. The latter adds 1,537 parameters and raises NaturalQuestions exact match from 48.5 to 51.4. Both copy tokens from flat text. We copy node identities and never copy strings verbatim.

Other work constrains the design. Pointer-Generator Networks for Low-Resource MT: Don't Copy That! shows that unconstrained copying pastes untransformed source tokens into output. This is exactly the failure a lemma-copying graph pointer would reproduce, and it is why pointing and realization are separated here. Older AMR realizers used tree transducers (Generation from AMR using Tree Transducers) and rules plus a language model to restore determiners and inflection (A Partially Rule-Based Approach to AMR Generation). They solved realization symbolically but gave no neural fluency. We keep their insight that realization is a distinct, constrainable step. Think Fast, Talk Smart routes data bindings through deterministic code and uses the neural model only for connectives. Our design is a learned, differentiable version of that partition.

The closest competitors are faithfulness methods for graph-to-text generation. FactSpotter reranks beam candidates with a classifier over triple–sentence pairs and admits it cannot judge multi-triple subgraphs. PlanGTG fine-tunes LLMs to reorder triples and emit attribution indices, cutting human-judged hallucinations from 0.84 to 0.36 per pair. Both are soft: the decoder can still name an entity that is not in the graph, or emit a tag that does not match its words. Our constraint is hard for entities and structural for relations.

Scene-graph captioners also bear on this. Consensus Graph Representation Learning reduces CHAIR by 9% by aligning visual and language graphs. Scene Graph Decomposition decodes from sub-graphs with soft node attention. Scene Graph Generation for Better Image Captioning? finds that a plain node-attention LSTM beats heavier graph encoders. These papers tell us the structure should enter through cheap pointing, not through heavy alignment machinery. None of them performs discrete pointing, and none models vague reference.

Research Program

The decoder runs on a pretrained encoder–decoder: T5-base, for comparability with FactSpotter, with a 7B variant for comparability with PlanGTG. Input graphs are linearized so that each node has a marked span. Node representations are pooled from those spans, with a light relational attention layer over edges. At each step a gate chooses between two actions. The first is generating from a restricted vocabulary of function words, punctuation and relation-phrase subwords. The second is pointing to a node $v$. The pointer score adds a learned bias $b(d)$ to the cross-attention score, where $d$ is the graph distance from the previously pointed node. This encourages coherent traversal without forbidding jumps.

After pointing, the decoder emits a surface form through trie-constrained decoding over a realization set $R(v)$. For WebNLG and DART, $R(v)$ contains the readable entity name, number and possessive variants, pronouns agreeing with the ontology type, and that type as a hypernym ("the astronaut"). For scene graphs it contains lemma inflections, articles and WordNet hypernyms. Relation phrases stay free-form but are generated only while the decoder is "on" an edge whose endpoints have both been pointed to. That scoping lets a node-conditioned verifier check each phrase against a specific edge.

Training needs node-level alignment for each mention. We obtain it by string and lemma matching plus coreference over references, and treat the alignments as latent where matching fails. A parallel analysis track uses the trace to score faithfulness. A mention counts as hallucinated only if its node is absent from the input. It counts as over-vague only if a human judges the pronoun or hypernym ambiguous in context. This scoring is compared against CHAIR and FactSpotter judgments.

Baselines

The naive reference point is T5-base fine-tuned on linearized triples, which FactSpotter reports at 93.60 FactS on DART. The strongest existing decoding-time method is the FactSpotter plug-in on the same T5-base, reported at 95.14 FactS on DART. It is the strongest competitor because it targets exactly the entity and predicate errors we aim to eliminate, and it requires no architectural change. We run it with the same beam size and training data. Beating it means exceeding 95.14 FactS while matching its BLEU within one point. We will also report FactSpotter-reranking applied on top of our decoder, since the two are complementary.

At LLM scale, the competitor is PlanGTG fine-tuning on Mistral-7B with its 30k instruction pairs. Its human hallucination rate is 0.36 per pair, versus TEKGEN's 0.84. We apply our decoder head to the same backbone and data, and run one shared human evaluation protocol for all systems. Beating it means a hallucination rate clearly below 0.36 under that protocol with no drop in human-rated fluency. A training-free competitor is LargePiG applied to the linearized graph with the same frozen backbone. It tests whether attention-derived copying alone suffices.

For captioning from gold Visual Genome scene graphs, we compare against G-LSTM+att, which reports 23.87 METEOR and 20.70 BLEU-4 on the COCO/Visual Genome split. We also compare against a flat token pointer-generator over the same linearized graph.

Four ablations isolate the key ingredients. One replaces the hard constraint with a soft pointer mixture. One removes the graph-distance bias. One shrinks $R(v)$ to exact lemmas, removing pronouns and hypernyms. One uses a pipeline that copies lemmas and then inflects them with a separate morphological model. Every variant gets the same tuning budget: equal hyperparameter sweeps and equal epochs.

Why not just…?

Why not just have the LLM emit node-ID tags, as PlanGTG does? This is the most serious alternative. A tagging model gives a trace for free, and if its tags were reliable our pointer would add little. The difference is enforcement. A tag is one more generated token, so the model can write a name absent from the input and attach a plausible index to it. Our experiment measures tag–text disagreement directly: the rate at which a PlanGTG mention's tag points to a triple that does not contain that entity. If that rate and PlanGTG's overall hallucination rate are already near zero on WebNLG 2020 and DART, the hard constraint is not worth its fluency cost, and we would report that.

Why not just prompt a strong LLM with the linearized graph and rerank? Fluent LLM output hides error rates of 76–86% on QUINTD-1 (Beyond Traditional Benchmarks). Few-shot prompts also leak demonstration entities into outputs (Kasner et al.), a failure our constraint excludes by design. Reranking can only choose among candidates that may all be wrong. We test this on QUINTD-1, where out-of-domain entities are exactly where soft methods should degrade.

Why not copy lemmas and fix morphology afterwards? A post-hoc inflector cannot choose between "Alan Bean", "he" and "the astronaut", because that choice depends on discourse. Inflection fixes form, but underspecification also involves reference. The pipeline ablation measures this directly, reporting grammaticality and human-rated naturalness of repeated mentions. If the pipeline matches us on both, the learned realization step is unnecessary.

Proposed Experiments

The decisive test on WebNLG 2020 and DART checks three things. Entity hallucination under our decoder should be zero by construction, which we verify with a human audit. Relation hallucination should fall below both the FactSpotter plug-in and PlanGTG. BLEU and human fluency should stay within a small margin. The last point is where the approach could fail, and the ablations will show whether any loss comes from the restricted vocabulary or from realization-set coverage.

On Visual Genome/COCO scene graphs, we generate from gold graphs to remove detector error and then from predicted graphs. We report METEOR against G-LSTM+att's 23.87, and CHAIR on the COCO Karpathy split, where Object Hallucination in Image Captioning reports rates of 7.4–17.7% for competitive models. We have no absolute CHAIR figure for the consensus-graph model beyond its reported 9% relative reduction.

The underspecification test uses Flickr30k Entities, whose phrase-to-box links let us check each pronoun or hypernym our decoder emits against its intended node. It asks two things. First, does trace-based scoring agree with human correctness judgments better than CLIPScore does on vague-but-correct captions? We will replicate the 100-pair stripping protocol from Dealing with Semantic Underspecification. Second, does enabling pronouns and hypernyms raise human naturalness without raising the ambiguity rate?

No reported results were found on Lacuna for pointer-based decoders on WebNLG or DART. The FactSpotter and PlanGTG numbers above are therefore the reference points.