Programs that write sentences about tables, such as "method A beats the best baseline by 3 points on every dataset except one," fail most often in technical domains. On scientific tables, the strongest pre-trained generators produce only about 39% factually correct statements (SciGen).
The best-known remedy routes generation through an executable logical form, a small program over the table such as argmax(filter(rows, dataset≠X), score). The program can be run to confirm that the claim is true before any text is written (Automatic Logical Forms improve fidelity in Table-to-Text generation). That remedy has two weaknesses that get worse as tables become more technical. First, the program languages used so far are small. They cover counts, superlatives and comparisons, but not the medians, ratios, exceptions and "for every" claims that scientific prose relies on. Second, executing the program only proves the program is true. It does not prove the sentence says what the program says. A realizer that writes "greater" for a < node passes every existing check.
We propose to close the second hole so that the first can be opened safely. After the system writes a sentence, an independently trained parser reads the sentence back into a program. The system accepts the sentence only if the read-back program behaves exactly like the original. "Behaves exactly like" is tested by running both programs on the real table and on many randomly perturbed copies of it. The same perturbation test also prunes programs that are true only by coincidence when we build training data. We then use it to build the training data for parser and writer from in-domain scientific tables rather than Wikipedia. That lets us measure, at fixed data size, whether domain-specific pre-training matters for this kind of structured reasoning or only for vocabulary.
Core Claim
The claim is that richer logic only improves faithfulness when it is paired with a check on the sentence itself. Once the program language is expressive enough to cover what scientific authors actually write, the dominant error moves from false programs to unfaithful verbalization of true programs. Execution-based rejection of false programs (False Candidate Rejection, or FCR, in TℓT) cannot see that error. A round-trip check judged by execution on perturbed tables can.
We expect two results. Without the round trip, fidelity on SciGen first rises and then stalls or falls as the grammar grows. With the round trip, fidelity keeps rising along with grammar coverage, at a measurable cost in how many tables receive any accepted statement.
The check is defined as follows. Let $P$ be the content program, $T$ the generated sentence, and $P'=\text{parse}(T)$. Let $\tilde{D}_1,\dots,\tilde{D}_K$ be perturbed copies of the input table $D$, with numeric cells resampled within their column's range and row order shuffled. We accept $T$ when
$$ \text{exec}(P, D)=\text{True} \quad\text{and}\quad \text{exec}(P', \tilde{D}_k)=\text{exec}(P, \tilde{D}_k)\ \text{for all } k=0,\dots,K, $$
where $\tilde{D}_0=D$. Agreement on random perturbations is a cheap test of whether two programs mean the same thing. Two programs that differ only on rare tables can slip through. For the operators we use, that probability shrinks quickly with $K$ because perturbations flip comparisons and argmaxes often.
Prior Work and Where This Sits
Several papers motivate the gap. Logic2Text pairs Wikipedia tables with executable logical forms. It shows that factual correctness drops to about 20% without the logical form and exceeds 82% with a gold one, so the intermediate program is the lever. TℓT replaces gold programs with a learned parser plus FCR and reaches 75% human-rated fidelity against 44.9% for end-to-end generation. Its authors also report the residual failure we target: the GPT-2 writer inverts comparisons that the program got right. SciGen shows that scientific tables, with floating-point cells and multi-level headers, are where end-to-end models collapse.
Other work supplies the methods we reuse. PLOG pre-trains T5 and BART to map tables to programs using execution-guided sampling over open-domain tables. It notes that its seven logic types exclude medians and standard deviations, and that T5-large handles formal syntax better than BART. We reuse its sampling recipe, but with a larger grammar, in-domain tables, and the perturbation filter. Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs shows in question answering that adding meta-operators like for_each and argmax lifts grammar coverage on CSQA from 80% to 96.2% and execution accuracy from 70.6% to 75.6%. That is direct evidence that coverage caps accuracy. We bring the same coverage measurement to generation, where it has not been made. Wang et al. close a loop by reconstructing the input data from generated text. We close the loop at the level of an executable program judged by behavior, which is stricter than matching extracted records.
Several papers constrain the design. SORTIE orders entity computations with a dependency graph and prunes coincidental programs with weak supervision, improving semantic-parsing accuracy over PLOG and REASTAP by 5.7% to 11.5%. Its operators are flat relational primitives, so we treat it as the strongest symbolic competitor rather than a component. ReTAG argues that discrete programs are too rigid and injects reasoning through learned codebooks instead. Those codebooks cannot be executed, so they are outside the regime where any of our checks apply. We address its rigidity argument head-on through the coverage measurement below. RKF filters free-text reasoning chains with a GPT-4o-distilled classifier and reports gains of 1.4 SP-Acc and 0.7 NLI-Acc points on LogicNLG. Its authors concede the distilled filter trails GPT-4o, which is the weakness of non-executable verification. Think Fast, Talk Smart finds that handing comparison, ranking or attribution to an LLM raises numeric error from 0.7% to 16.9% and selection error to 65.8% in health reports. Its deterministic layers are handwritten, which is exactly what a learned, checked pipeline should replace.
On domain-specific pre-training, BioBART shows that continual pre-training on PubMed improves biomedical generation (ROUGE-L 53.15 vs. 51.05 on MeQSum). It also shows that a generic objective, sentence permutation, hurts in-domain, so the choice of pre-training objective matters. Männistö finds that domain pre-training predicts code-generation quality better than the choice of parameter-efficient tuning method. Neither tests pre-training on structured table-to-program data. Whether in-domain tables beat Wikipedia tables for this purpose is the question we isolate.
Research Program
We will build four things.
The first is a graded ladder of program languages. Level 0 is conjunctive filters only. Level 1 adds PLOG's seven types. Level 2 adds ratios and percentage differences, median and variance, negation and "all but one" exceptions. Level 3 adds universal and "most" quantifiers over row groups and multi-level header indexing. Level 4 is unrestricted pandas code. The upper levels address the expressivity gap. To measure the gap itself, two annotators will label a sample of SciGen expert-written reference statements with the lowest level that can express each one. This gives a coverage curve, the generation analogue of Marion et al.'s 80%-to-96.2% figure, which the file does not report for any data-to-text benchmark.
The second is synthetic pre-training data. We sample programs at each level over tables from SciGen's automatically extracted Medium and Large splits, excluding papers in any test split. We keep a program only if it executes to True on the real table and is not constant across perturbed tables, which removes trivially true and coincidental programs. We pair programs with text in two ways. Programs extracted from reference sentences give natural pairs. Template realizations give extra coverage. A matched-size corpus built the same way from Wikipedia tables serves as the domain control.
The third is two models trained independently. The table-to-program planner-and-writer is a T5-large continually pre-trained on table-to-program and program-to-text, then fine-tuned on SciGen. The text-to-program parser is a separately initialized model trained on the reverse direction with different data order and seeds, so that its errors are less correlated with the writer's.
The fourth is the selection loop. For each table we sample many candidate programs, apply FCR and then the round-trip check, and rank survivors with a salience scorer trained on which programs match reference statements. This avoids the oracle cell highlighting that TℓT and CONTLOG rely on.
Baselines
The naive reference point is a T5-large fine-tuned end-to-end on SciGen with linearized tables and no intermediate program. SciGen reports roughly 39% human-rated statement correctness in the best case for BART and T5. We rerun this baseline under our human protocol rather than relying on the reported figure.
The strongest existing symbolic method on our target benchmark is SORTIE. It was evaluated on SciGen, but no isolated SciGen number was found on Lacuna for SORTIE; only its aggregate 5.7% to 11.5% SP-Acc gain over PLOG and REASTAP is available. We will retrain it on the same SciGen training data with the same backbone size and report it under our human evaluation.
PLOG is the closest pre-training competitor. It reports about a 10% TAPEX-Acc gain over T5-large on LogicNLG and 79% human-assessed accuracy on CONTLOG. We run it with its own seven-type, Wikipedia-sampled corpus, then fine-tune on SciGen with the same steps and tuning budget as our model.
The TℓT-style pipeline (planner, FCR, writer, no round trip) uses our exact planner, writer and grammar. It is the ablation that isolates the key ingredient. TℓT reports 75% human fidelity on Logic2Text with oracle cell selection, and we will report its number on SciGen without oracle selection.
The code-writing LLM baseline prompts a strong instruction-tuned LLM to write pandas code over the table. The code is executed with FCR, and the LLM then verbalizes the result. It gets the same number of candidate samples per table as our system.
The text-level filter baseline replaces the round trip with a table-entailment classifier of the kind behind LogicNLG's NLI-Acc. It is trained on the same SciGen data.
Beating a baseline means higher human-rated correctness among accepted statements at matched coverage, where coverage is the fraction of tables with at least one accepted statement. We tune each system's acceptance threshold to the same coverage, then compare correctness with a paired bootstrap over tables. A gain that exists only at lower coverage is reported as a trade-off, not a win.
Why not just…?
Why not just have a large language model write Python and run it? This is the most serious threat, because pandas is maximally expressive and current LLMs write it well. FCR still checks only the code. Kasner et al. document few-shot leakage of values from prompt examples into outputs. Think Fast, Talk Smart shows that even an LLM-formatted handoff raises numeric error from 0.7% to 4.8%. Both are verbalization errors a code check cannot catch. The round trip is language-agnostic, so we also run it on top of the Python baseline (Level 4). If Python plus round trip matches our trained typed-grammar system, the honest conclusion is that the check is the contribution and the custom grammar and domain pre-training are not needed. The ladder experiment is designed to reveal exactly that.
Why not just filter generated sentences with an entailment classifier? Entailment classifiers are cheap and need no parser. However, they judge lexical and numeric plausibility rather than meaning, and RKF's experience is that learned filters trail the teacher and leave residual errors. Our direct comparison at matched coverage, broken down by error type, will show whether the classifier catches comparison inversions and wrong-quantifier claims as often as execution on perturbed tables does.
Why not just fine-tune a domain-pretrained text model and skip table-to-program pre-training? BioBART and Männistö show that domain text pre-training helps. Neither installs numerical reasoning, and SciGen's 39% comes from models that already had broad pre-training. We test this directly by fine-tuning a scientific-text continually pre-trained T5 with the round-trip pipeline but no synthetic table-program stage. We also compare in-domain versus Wikipedia synthetic corpora at equal size. If Wikipedia-sampled programs match arXiv-sampled ones, domain specificity is not what matters here, and we will report that.
Why not just expand PLOG's grammar and stop? The ladder answers this. If fidelity without the round trip keeps climbing with grammar level, the expressivity fix alone suffices and our check is unnecessary.
Proposed Experiments
The primary evaluation is SciGen's expert-annotated split. Human judges rate each accepted statement as correct, incorrect or unverifiable against the table. We also report coverage and the share of statements that are analytical rather than descriptive. BLEU is reported only for completeness, since n-gram overlap rewards contradictory statements. Logic2Text and LogicNLG serve as secondary benchmarks, so that results can be placed against PLOG, TℓT and DCVED. DCVED reports 30.8% human-rated fidelity against 19.1% for GPT-TabGen on LogicNLG.
Four checks decide the outcome.
The first is the main result. At matched coverage, round trip plus FCR must beat FCR alone on SciGen correctness, and the gain must concentrate in comparison-direction, quantifier and exception errors. That would confirm that verbalization is the remaining failure.
The second is the ladder. We plot reference-statement coverage and accepted-statement correctness against grammar level, with and without the round trip. The prediction is a plateau or decline without the round trip and continued gains with it.
The third is the domain control. Holding corpus size, grammar and compute fixed, arXiv-sampled pre-training must improve parser accuracy and final correctness over Wikipedia-sampled pre-training. Otherwise the domain-specific claim is dropped.
The fourth checks the check itself. On a few hundred hand-labeled sentence-program pairs, we measure how often the round trip wrongly rejects faithful sentences, how often it wrongly accepts unfaithful ones, and how both rates change with $K$ and with deliberately correlated parser and writer (shared initialization). This tells us whether independent training is necessary.
References
- SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables
- Automatic Logical Forms improve fidelity in Table-to-Text generation
- Logic2Text: High-Fidelity Natural Language Generation from Logical Forms
- PLOG: Table-to-Logic Pretraining for Logical Table-to-Text Generation
- Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs
- Faithful Low-Resource Data-to-Text Generation through Cycle Training
- SORTIE: Dependency-Aware Symbolic Reasoning for Logical Data-to-text Generation
- ReTAG: Reasoning Aware Table to Analytic Text Generation
- Reasoning Knowledge Filter for Logical Table-to-Text Generation
- Think Fast, Talk Smart: Partitioning Deterministic and Neural Computation for Structured Health Text Generation
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- A Comparative Study of PEFT Methods for Python Code Generation
- Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation