Entity-Blind Cycle Training for Data-to-Text Without Shared Entities

Data-to-text generation turns structured records, such as knowledge-graph triples or table rows, into sentences. Fluent systems still invent facts and drop required ones. One of the most effective low-resource fixes is cycle training. A generator writes text from data, an extractor recovers data from that text, and each model is trained to reconstruct what the other started from, so large amounts of unpaired data and text can be used. Cycle training has a known weakness: it breaks down when the unpaired data and the unpaired text mention mostly different entities. That is exactly the situation of a practitioner who has a database about one set of companies or athletes and a text corpus about others.

We propose to run the whole cycle over entity-blind representations. Before either model sees an example, every entity mention is replaced by a typed, indexed placeholder such as [PERSON_1] or [CITY_2]. The real names are put back only after generation. Our claim is that the overlap requirement is an artifact of reconstruction losses being spent on reproducing entity names, not a limit of cycle consistency itself. Once names are hidden, both reconstruction losses can only be reduced by getting the relations and predicates right. That supervision transfers across disjoint entity vocabularies, and it targets the relational errors that entity-level faithfulness methods leave unmodeled.

Where This Sits

The method comes from Faithful Low-Resource Data-to-Text Generation through Cycle Training. That paper pairs a data-to-text generator with a text-to-data extractor in Data-Text-Data and Text-Data-Text loops. With only 100 paired WebNLG samples it reports PARENT 46.14, against 45.67 for full supervision. PARENT is a table-aware overlap score that rewards content supported by the input. On WikiTableQuestions it cuts factual errors from 14.71 per triple to 0.00. The same paper also supplies the motivating negative result: cycle training fails when entity overlap between the two unpaired corpora drops below about 50%, and it operates on raw strings throughout. We keep its training loop and change only what flows through it.

The masking ingredient comes from the entity-blinding tradition summarized in the file: anonymized reading comprehension (Hermann et al.), type masking for relation extraction (Chen), and zero-coverage mention randomization for named-entity recognition (Lin et al.). Each of these was used to stop classifiers from memorizing names. None was applied to a generate-and-extract cycle. Injecting Entity Types into Entity-Guided Text Generation shows that type information, not surface names, guides the choice of surrounding verbs. That result motivates typed rather than purely random placeholders, though its experiments were on news text with no structured input to check against.

Three entity-centric faithfulness papers constrain the design. Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View trains a generator to follow entity plans and filters training data to the cleanest 5%. Its plans keep verbatim names, and it needs paired data. Entity-Based Semantic Adequacy for Data-to-Text Generation gives us an evaluation tool, entity-based semantic adequacy (ESA), that detects missing input entities. It also warns that automatic entity detection is imperfect: precision 0.75 and recall 0.74. That matters because our masking of unpaired text relies on such a detector. Sundararajan et al., 2024 show that suppressing hallucination through stricter inputs raises omissions. We therefore track omissions as a first-class outcome, not a side metric.

Core Claim

With typed, indexed entity placeholders, cycle training keeps its low-resource faithfulness gains when the unpaired data and unpaired text share few or no entities. We expect the advantage over raw-string cycle training to grow as overlap falls. The mechanism we propose is checkable. Under raw strings, most of the reconstruction loss falls on entity tokens. When entities do not overlap, that loss cannot be reduced by learning anything shared between the two corpora. Masking moves the loss onto predicate and relation tokens, which the two corpora do share.

Research Program

The pipeline has three parts.

The first is a masking step applied identically to both sides. On the data side, subjects and objects of triples, or the values of entity-bearing table cells, are replaced by placeholders. Each placeholder is typed from the source ontology or column header and indexed within the example. On the text side, an entity detector with a type tagger does the same. Placeholders for one example are linked consistently: the same entity gets the same index in the data and in the text. Numbers and dates are left raw in the main condition and masked in an ablation, because numeric copying behaves differently from entity naming.

The second is cycle training as in Wang et al., unchanged except that both models read and write masked sequences. Data-Text-Data reconstruction now asks whether the extractor recovered ([PERSON_1], birthPlace, [CITY_1]), not whether it spelled a name correctly.

The third is deterministic re-lexicalization. Each placeholder is replaced by its source surface form, with a small rule set for English possessives and articles. Placeholders that the generator emits with no counterpart in the input count as hallucinations by construction. Input placeholders that never appear in the output count as omissions by construction. Because each output is checked against its own placeholder set, this check needs no entity detector on the output.

We will compare two placeholder schemes. Typed indexed tags carry the coarse semantic class. Randomized names, following Lin et al., keep text natural-looking for the pretrained language model but carry no type. This comparison tests the granularity trade-off the masking literature flags as open: whether type information helps predicate selection more than natural-looking strings help fluency.

For the mechanism analysis, we decompose the reconstruction loss by token class (entity, predicate, function word) under raw and masked training, across overlap levels.

Baselines

The naive reference point is supervised fine-tuning of the same pretrained encoder-decoder on the small paired set alone, on raw strings. This is the condition Wang et al. report as producing 14.71 factual errors per triple on WikiTableQuestions. It tells us how much any use of unpaired data buys.

The strongest existing method is raw-string cycle training as in Wang et al. It is the exact system whose failure we claim to fix, and it is reported to beat full supervision on WebNLG. The number to match is its reported PARENT of 46.14 with 100 paired samples. That figure comes from Wang et al.'s high-overlap setting. We found no reported result on Lacuna for cycle training below the 50% overlap threshold, so we will rerun it ourselves at every overlap level. We will reuse the same backbone, the same paired seed set, the same unpaired pools, the same number of cycle iterations, and the same hyperparameter search budget for both raw and masked cycles.

We will also run a raw cycle with a soft copy switch, in the style of TWT. This tests whether the gain comes simply from making entity names easy to reproduce.

The ablation that isolates our key ingredient is masked supervised fine-tuning without the cycle. It uses the same placeholders and the same seed pairs but no unpaired data. If masked cycle training does not clearly beat this, the cycle is not contributing.

Beating the strongest baseline means two things. At 50% overlap or less, masked cycle training must achieve higher PARENT and lower hallucination than raw cycle training. At high overlap, it must not fall measurably below raw cycle training. All of this must hold without an omission increase larger than the hallucination decrease.

Why not just…?

Why not just fine-tune or prompt a larger pretrained model? Scale has not closed the gap. Kasner and Dusek (2024) find semantic errors in 76% to 86% of zero-shot open-LLM outputs on QUINTD-1. In few-shot prompting they also find that entities bleed from the demonstrations into the target output. Parikh et al. (2020) report 76.2% faithfulness for a fine-tuned BERT-to-BERT model on ToTTo, against 93.6% for humans. We will still include a few-shot open-LLM row on our WebNLG splits, so the comparison is measured rather than assumed.

Why not just delexicalize and fine-tune on the paired data, with no cycle? This is the real threat. Delexicalization is old, and on templated RDF data it may capture most of the benefit by itself. The masked-supervised ablation answers it directly. If that ablation matches masked cycle training, we will report that the overlap rule is fixed by masking alone. The contribution would then shrink to a diagnosis plus a simpler recipe. We would accept that outcome rather than argue around it.

Why not just collect unpaired text whose entities overlap the data? In the target use cases, such text does not exist: new entities, private databases, or text from a neighbouring domain. Filtering also discards coverage. Liu et al. keep only 5% of their data. Our overlap sweep reports where raw cycles work, so a practitioner who can obtain overlapping text can tell whether masking is needed.

Why not just add a copy mechanism? Copying fixes spelling, not relations. Dong et al., 2021 report that type-injected generation beats copy-based baselines on coherence. Parikh et al. note that pointer models still hallucinate comparatives such as "first." The copy-enabled raw cycle baseline tests this alternative under identical training.

Proposed Experiments

The main testbed is WebNLG, using its 16 DBpedia categories to build controlled overlap splits. The paired seed set holds 100 examples. The unpaired triples and unpaired texts are drawn so that their entity sets overlap at 0%, 25%, 50%, 75%, and 100%. We build these within categories, and also across categories for a harder domain-shift condition. We then repeat the overlap sweep on WikiTableQuestions, WikiSQL, and E2E, following the low-resource cycle settings of Wang et al. ToTTo serves as a check that the method holds on single-sentence table descriptions.

The metrics are of three kinds. Fluency and grounding are measured by PARENT and BLEU. Entity faithfulness is measured three ways: by ESA, by the placeholder-level hallucination and omission counts described above, and by $P_{cover}$ and $R_{hallu}$ in the style of Liu et al., where $P_{cover}$ is the share of input records realized in the output and $R_{hallu}$ is the share of output entities absent from the input. Finally, a human evaluation on 300 outputs per system labels predicate errors separately from entity errors, because predicate faithfulness is what ESA leaves unmeasured.

The decisive checks are four. The first is whether the raw-cycle curve drops below the 50% overlap point while the masked-cycle curve stays flat. The second is whether masked cycle training beats masked supervised fine-tuning, which shows the cycle matters. The third is whether the share of reconstruction loss falling on predicate tokens rises under masking, which confirms the mechanism. The fourth is whether gains survive realistic masking noise. For this we deliberately degrade the text-side entity detector toward the roughly 0.75 precision and recall level reported by Faille et al. We will also report re-lexicalization grammar errors from human judgments, since restoring names after masking is a known weak point of the approach.