Language models now generate synthetic tables, such as patient records, loan applications, and housing listings, by writing each row as a short piece of text one token at a time. They still produce impossible rows. A diastolic pressure ends up above the systolic, or a coordinate pair lands in the wrong state. The cheap fix is to forbid bad tokens during generation, which guarantees valid rows. But that fix quietly changes which valid rows come out: forbidding a token at one step pushes probability onto whatever valid tokens are left, so rows whose prefixes the model "almost got wrong" are over-sampled. This note proposes to measure that distortion and remove it at training time. The model is trained so that, at every step, it already puts essentially no probability on tokens that would make the row impossible. Once that holds, forbidding tokens changes nothing, and fast constrained decoding samples exactly from the model's distribution restricted to valid rows.
The opportunity sits between two literatures that have not met. Constraint enforcement for tabular generators, by projection layers, penalty losses, or symbolic filters, has been built almost entirely for GANs, VAEs and diffusion models. Language-model-based table generation, along with the broader habit of adapting language models to a domain by further pre-training on in-domain data, has attacked violations only through better column ordering or more domain text. The latter does not suffice: a biomedical pre-trained model still broke clinical consistency rules in Critic–Adviser–Reviser Cyclic Refinement. We connect the two by turning a compiled constraint set into a per-step feasibility check and using it twice, once as a hard mask at decoding and once as a training signal during domain-adaptive continued training.
Core Claim
Write a serialized row as tokens $x_1,\dots,x_T$. Let $F_t$ be the set of next tokens after prefix $x_{<t}$ that still admit some valid completion, and let $Z_t = \sum_{v\in F_t} p(v\mid x_{<t})$ be the probability mass the model puts on them. Masked decoding samples from $q(x)=\prod_t p(x_t\mid x_{<t})\,\mathbb{1}[x_t\in F_t]/Z_t$. The ideal target is the model restricted to the valid set $C$, namely $p(x\mid C)=p(x)\mathbb{1}[x\in C]/P(C)$. For any valid row,
$$\frac{q(x)}{p(x\mid C)} = \frac{P(C)}{\prod_{t} Z_t(x_{<t})},$$
so masking is unbiased exactly when $\prod_t Z_t$ is constant across valid rows. It is trivially unbiased when every $Z_t$ equals 1. The claim is that a training loss which drives $Z_t$ toward 1 on the prefixes the model actually visits will produce constrained samples that match rejection sampling (which is exact for $p(x\mid C)$) in fidelity, at the cost of masked decoding. This also gives a diagnostic: the spread of $\prod_t Z_t$ over generated rows tells us, before any downstream evaluation, how much a masked sampler is distorting the distribution.
The novelty is not "fewer violations." Masking already gives zero. The novelty is closing the fidelity gap between cheap hard enforcement and exact conditional sampling, by teaching the model the constraint structure rather than patching its outputs.
Research Program
The feasibility check reuses an existing tool. The Disjunctive Refinement Layer of Beyond the convexity assumption compiles quantifier-free linear arithmetic constraints, including disjunctions, into a sequence of bounds on each feature given the earlier ones. That paper uses these bounds to project GAN outputs onto the nearest valid point. We use them differently: because the compilation guarantees that any value inside the bound for feature $i$ can be completed validly, the bounds are exactly a prefix feasibility oracle for left-to-right generation. Numbers are tokenized as digits, so a digit token is feasible if some continuation of the partial number falls inside the current interval, which is simple interval arithmetic. Discrete functional dependencies, such as Zip Code determining State, are handled by lookup.
Column order matters because the compiled bounds depend on the elimination order. We adopt the dependency-graph topological ordering of PAFT (Are LLMs Naturally Good at Synthetic Tabular Data Generation?), which puts causes before effects, and check that DRL compilation stays tractable under that order.
Training adds a semantic-loss term of the kind surveyed in Deep Learning with Logical Constraints. That paper's caveat that such losses give only soft guarantees is exactly why we keep the hard mask at decoding. The objective is
$$\mathcal{L} = \mathcal{L}_{\text{NLL}} + \lambda\,\mathbb{E}_{x_{<t}\sim \text{model}}\big[-\log Z_t(x_{<t})\big].$$
The key design choice is that the second expectation runs over prefixes sampled from the model itself, not only over real-data prefixes. On real rows the correct next token is always feasible, so the likelihood term already pushes some mass toward $F_t$. The distortion, however, arises on the slightly-off prefixes the model visits during generation, and only on-policy prefixes reach those.
The domain dimension enters as continued pre-training on a pool of tables from one domain. The clinical pool uses WIDS and Cervical Cancer Risk; the finance pool uses HELOC and LendingClub. Each table carries its own compiled oracle. Column names are anonymized, following MonoCoder's (MonoCoder) removal of superficial local names. The point is to test whether constraint-respecting behavior transfers within a domain or is just memorized per table. This guards against the failure exposed in The Illusion of Generalization in Tabular Language Models, where Tabula-8B's apparent gains came from column-name shortcuts and contamination.
Constraint sets come from the benchmark ground truth: 13 constraints on House, 12 on CCS, 31 on WIDS. For WIDS we also try constraints selected by the EVS pipeline of From Noisy Oracles to Useful Constraints, which showed that unverified LLM-proposed rules can cut AUC by up to 26 points.
Baselines
The naive reference point is a PAFT-ordered language model fine-tuned on each table with plain likelihood and sampled without constraints. It shows the raw violation rate and the fidelity we are trying to preserve. The cited figure is a violation rate of 3% for PAFT on US-locations, versus about 22% for GReaT's ordering, both from Are LLMs Naturally Good at Synthetic Tabular Data Generation?.
The strongest language-model competitor is the same PAFT model with rejection sampling. It is exact for $p(x\mid C)$ and is therefore the fidelity ceiling for an untouched model. Its weakness is cost, which grows with the invalid mass. We will report cost per valid row alongside fidelity. Beating it means matching its fidelity within noise at a small fraction of its cost, or exceeding its fidelity if the extra loss also improves $p(x\mid C)$ itself.
The PAFT model with masked decoding but no training change is the ablation that isolates our ingredient. If masking alone already matches rejection-sampling fidelity, the distortion we posit is negligible and the training loss has nothing to fix.
A second ablation applies the feasibility loss only on real-data prefixes, which tests whether sampling prefixes from the model is necessary.
The strongest non-language-model competitor is DRL attached to CTGAN, TVAE and GOGGLE. This is a single-pass generator with 0% violations across House, CCS, HELOC, LendingClub and URL, and F1 lifts of up to 21.4% on CCS, all reported in Beyond the convexity assumption. We found no reported result on Lacuna for a language-model generator combined with DRL-style bounds on these tables.
Every language-model arm uses the same base model, data splits, token budget for continued training, and the same hyperparameter search budget. The GAN arms use the original DRL configurations.
Why not just…?
Why not just reject invalid samples? For a model that already violates only 3% of the time, rejection is cheap and exact, and it is a real threat on such tables. It breaks down where invalid mass is large. DRL's authors report that rejection can take hours per valid batch when base violation rates approach 100% on House (Beyond the convexity assumption), and stacking constraints causes heavy sample loss (CuTS). Our experiments plot fidelity against cost per valid row across tables with low and high invalid mass. If rejection is affordable everywhere for language-model generators, we will say so, and the contribution shrinks to the diagnostic.
Why not just mask invalid tokens at decoding? Masking guarantees validity, and it is our own decoding procedure. The equation above shows its bias is governed by how unevenly the model's invalid mass is spread across rows, and nothing in the file measures that bias for tabular language models. The mask-only ablation tests this directly. If the importance-weight spread is near zero and fidelity matches rejection sampling, the training term is unnecessary.
Why not just use DRL on a GAN or VAE, which already gives 0% violations in one pass? On purely numeric linear-constraint tables like House and HELOC this may well win. We treat it as the bar, not a straw man. The case for language models rests on mixed categorical tables, where PAFT beat CTGAN (99% violations on US-locations). The comparison will show on which kinds of tables each approach is preferable.
Why not just continue pre-training on more domain text? A biomedical pre-trained model still failed consistency rules without external refinement (Critic–Adviser–Reviser Cyclic Refinement). The domain-pool arm trained without the feasibility loss is included to confirm this in the tabular setting.
Proposed Experiments
The main experiment runs every arm on House, CCS, HELOC, LendingClub, URL, WIDS and US-locations. For each arm we report:
- violation rate;
- train-on-synthetic, test-on-real F1 and AUC;
- discriminator "sniff test" accuracy, where closer to 50% is better;
- pairwise column-dependence error;
- generation cost per valid row;
- the importance-weight diagnostic, computed as the effective sample size of masked samples relative to $p(x\mid C)$ from the spread of $\prod_t Z_t$.
The first decisive check is whether, under masked decoding, the feasibility-trained model reaches the fidelity of rejection sampling while the mask-only model does not, with the gap growing with the untrained model's invalid mass. If there is no gap, the central claim is refuted.
The second check is whether the on-policy loss beats the data-prefix-only loss. This tests the proposed mechanism and not just the presence of an extra regularizer.
The third check is the within-domain transfer test. After continued training on the clinical or finance pool with anonymized columns, we measure the average $Z_t$ on a held-out table from the same domain before any per-table fine-tuning. We compare a pool trained with the loss against one trained without it, and expect only modest transfer. We will also check whether any improvement survives a contamination audit of the kind in The Illusion of Generalization in Tabular Language Models.
References
- Critic–Adviser–Reviser Cyclic Refinement: Towards High-Quality EMR Corpus Generation with LLMs
- Beyond the convexity assumption: Realistic tabular data generation under quantifier-free real linear constraints
- Why LLMs Are Bad at Synthetic Table Generation (and what to do about it)
- Deep Learning with Logical Constraints
- MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks
- The Illusion of Generalization in Tabular Language Models
- From Noisy Oracles to Useful Constraints: LLM-Guided Constraint Selection for Synthetic Tabular Data
- CuTS: Customizable Tabular Synthetic Data Generation