Large language models are increasingly used to clean spreadsheets and databases. They fix typos, fill missing cells, and resolve values that break rules such as "each zip code belongs to one state." Nearly every table used to evaluate them is public: hospital records, flight logs, food inspections, loan data. The clean versions of those tables are very likely in the models' pre-training data. When such a model "repairs" a cell, we cannot currently tell which of three things happened. It may have reasoned from evidence in the table. It may have recalled the value from the memorized public original. Or it may have invented a plausible value, which is a hallucination.
This proposal builds counterfactual twin tables. A twin is a copy of a public table in which entities and values are changed consistently, so that every rule and relationship in the table still holds but the memorized answer is now wrong. We inject identical errors into the original and the twin. The gap in repair quality between the two then measures how much apparent cleaning skill is memory. We then use the same twin trick at inference time as a gate. A repair is accepted only if it changes appropriately when the supporting evidence in the table is changed. Otherwise the cleaner abstains.
Core Claim
The claim has two parts. First, reported accuracy of LLM-based table cleaners on public benchmarks overstates their ability to repair unseen data. The overstatement grows under exactly the conditions where cleaning matters most: heavy corruption and noisy retrieved context. Under those conditions the in-table evidence is weak, and the model falls back on memory or fabrication. Second, a repair that ignores evidence can be detected cheaply without any knowledge of the training corpus. Swap the evidence for a consistent counterfactual and check whether the proposed value follows it. This per-repair evidence-tracking test should flag both memorized repairs and fabricated repairs better than generic uncertainty signals do.
Relation to Prior Work
Three lines of work motivate the gap.
- LLM cleaners on public tables. AutoDCWorkflow has an LLM write OpenRefine cleaning operations for 96 public tables. The file notes it does not test whether success comes from memorized schemas, and that it degrades and invents operations under heavy noise. RetClean grounds cell repair in tuples retrieved from a data lake. It assumes that data lake is clean and never audits stand-alone model accuracy for leakage. Our twins address the first gap directly. Polluted-lake experiments address the second.
- Contamination detection. The contamination survey catalogues detection by n-gram matching, membership inference, and chronology. All of these operate at the level of a whole dataset or item. None says whether a particular repair came from memory, which is what a cleaning system needs at run time.
- Answers copied from retrieved context. Search-Time Data Contamination shows search agents copying leaked answer keys from retrieved pages. It is the retrieval-side analogue of our concern, but it offers no filter. RetClean-style lakes are exactly the local setting it leaves unexamined.
Two papers supply the method.
- Perturbing facts. INVITE distorts DBpedia triples within a semantic category and verifies the result does not exist, producing false-premise questions. We borrow this within-category, verified-nonexistent perturbation. We extend it from single facts to whole relational tables, where the perturbation must preserve functional dependencies across rows.
- Procedural generation against contamination. DyVal avoids contamination by generating reasoning problems procedurally. It is limited to formal tasks. Twins keep real table structure and real error patterns instead of synthesizing from scratch.
Two further lines constrain the design.
- Noisy context degrades models. Cho et al. and UDA show readers degrade as irrelevant context grows. In UDA, GPT-4 falls from 45.9 to 0.4 on FinHybrid without retrieval. So we must vary how many dirty neighboring rows the model sees, not assume more context helps.
- Errors interact. Abedjan et al. and UniClean establish that errors interact and that no single cleaner wins everywhere. Error injection must therefore be mixed (typos, rule violations, missing values) and applied identically across twins.
Research Program
Twin construction. For a clean public table with known rules, we build a consistent bijective renaming of entity strings within each semantic type. City maps to a real but different city, and person names map to nonexistent but well-formed names, verified against the original table and common knowledge sources as INVITE does. Numeric columns get order- and dependency-preserving transformations, such as consistent offsets per key. We then re-check that every functional or conditional dependency still holds. Multiple twins per table give variance estimates.
An automatic check guards against twins that are simply unnatural. A classifier that can tell twin rows from original rows at near-chance level indicates the twin is not artificially odd. A small human plausibility audit backs this up.
Error injection. The same seeded mix of typos, format variants (in the spirit of DataVinci), rule violations, and deletions is applied at matched positions in both the original and the twin. Corruption rate is swept from light to heavy.
Scoring outcomes. On a twin, each repair falls into one of four categories:
- correct: matches the twin's clean value;
- memorized: matches the original public value;
- fabricated: matches neither;
- abstained.
The memorized category is exactly identifiable only on twins, and this is the benchmark's main contribution.
The gate. For each proposed repair, we locate the supporting evidence the model could have used. This includes rows sharing the same key and values tied to the cell by the table's rules. We build a small local counterfactual by consistently renaming that evidence, then re-query. An evidence-grounded repair should move to the correspondingly renamed value. A memorized repair stays put, and a fabricated one moves erratically. The repair is accepted only if it tracks the evidence across two counterfactuals. The cost is a small constant number of extra queries per flagged cell, not per table.
Noise sweeps. We vary two things. The first is the number of dirty neighbor rows shown in the prompt. The second, for retrieval-based cleaners, is the fraction of corrupted or contradictory tuples in the data lake. This traces how the balance of correct, memorized, and fabricated repairs shifts as evidence degrades.
Baselines
Naive reference. A no-repair baseline, which leaves dirty cells unchanged, is scored on the same metrics. It anchors precision and recall and shows whether any system helps on heavily corrupted twins.
Rule- and statistics-based cleaners. The strongest non-LLM competitor is UniClean (UniClean), which combines rule-based, distribution-based, and knowledge-enhanced cleaners in an optimized workflow. We also run NADEEF (NADEEF). These methods rely mostly on the table's own rules and statistics rather than on a language model's recall, so their twin gap should be comparatively small. That makes them the natural control for whether LLM gains survive contamination removal.
LLM cleaners. The strongest LLM competitors are RetClean (RetClean) in its stand-alone and retrieval modes, and AutoDCWorkflow (AutoDCWorkflow). The file contains no reported result on Lacuna for RetClean, AutoDCWorkflow, or UniClean repair accuracy. We will therefore re-run all three ourselves on identical dirty originals and twins, with the same backbone model where an LLM is involved, the same prompt budget, and the same number of tuning trials on a shared validation split drawn from different tables.
Alternative gates. The strongest existing way to decide when to trust a generation is noise-enhanced sampling from Enhancing Hallucination Detection through Noise Injection. It injects noise into upper-layer activations and measures output entropy across samples. We give it the same number of extra forward passes as our gate, and tune its noise magnitude on the validation split. A plain prompted-abstention gate ("answer UNKNOWN if not inferable from the table") serves as the cheapest alternative.
Key ablation. To isolate the counterfactual ingredient, we compare against a resampling gate. It re-queries the same number of times on the unchanged table and accepts a repair only if it is stable. This separates the benefit of changing the evidence from the benefit of simply asking again.
What counts as beating a baseline. The gated LLM cleaner must match or beat the best baseline's repair F1 on twins. It must also cut the memorized-plus-fabricated share of accepted repairs, with non-overlapping confidence intervals across twin seeds.
Why not just…?
Why not just evaluate on fresh, post-cutoff tables, as LiveBench does? LiveBench limits contamination by rotating in new material. Fresh tables do lower memorization, but they change the table distribution at the same time. A drop in score cannot then be attributed to memory rather than difficulty. They also cannot label an individual repair as memorized. Twins hold structure and error positions fixed and change only whether memory helps. We will still include a set of post-cutoff tables as a sanity check. If the twin gap and the fresh-table gap disagree badly, that is evidence our twins are unnatural.
Why not just run a membership-inference or n-gram contamination detector? Such detectors, surveyed in the BDC survey, answer "was this table seen?" A cleaner deployed on private data needs "is this repair supported?" That question can have a bad answer even for never-seen tables, because models fabricate. INVITE reports open-source models exceeding 80% hallucination on distorted facts, against 6% for GPT-4. We will report detector scores alongside twin gaps to show whether table-level flags predict per-repair memorization.
Why not just skip LLMs and use constraint-based cleaners? This is a real threat. If UniClean matches gated LLM cleaners on twins, the honest conclusion is that LLM cleaning gains on public benchmarks were largely memory. The twin experiment settles this directly. We expect LLMs to retain an edge only on semantic string errors that rules cannot express, and we will report per-error-type results so that claim can fail.
Why not just gate with noise-injection uncertainty? Memorized values are typically produced confidently. Entropy-based gates should therefore miss exactly the memorized category, while possibly catching fabrications. That paper also notes its noise magnitude must be tuned per model family. The per-category breakdown (memorized versus fabricated caught) will show whether our gate adds anything beyond this baseline.
Proposed Experiments
Tables and error sources.
- AutoDCWorkflow's six domains: Hospital, Flights, CFI, PPP, Menu, Dish.
- RetClean's Cricket Players, QCRI Personnel, and Animals, with their data lakes.
- A string-repair slice following DataVinci's error types.
- Pollock's polluted CSVs (Pollock) to add loading-level corruption upstream.
- UDA's FinHybrid for a downstream check: whether gated repair of parsed tables improves answer accuracy.
Metrics.
- Repair precision, recall, and F1.
- The four-way outcome shares on twins.
- The twin gap, defined as original F1 minus twin F1.
- Abstention precision, meaning the fraction of abstained cells that the ungated model would have gotten wrong.
Decisive checks. Three results would settle the proposal.
- Is there a memory effect? Ungated LLM cleaners should show a twin gap that grows with corruption rate, while UniClean's stays small. A flat gap would falsify the first claim.
- Does evidence tracking matter? The counterfactual gate should reduce memorized repairs among accepted outputs more than the resampling ablation and the noise-injection gate at equal query budget. If resampling does as well, the counterfactual ingredient is unnecessary and we will say so.
- Is RetClean's clean-lake assumption fragile? In the polluted-lake sweep, ungated retrieval cleaners should copy corrupted lake tuples at a rate rising with pollution, mirroring search-time copying. The gate should catch these copies, because corrupted evidence does not track the counterfactual renaming of the in-table evidence.
References
- AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmark
- RetClean: Retrieval-Based Tabular Data Cleaning Using LLMs and Data Lakes
- Benchmark Data Contamination of Large Language Models: A Survey
- Search-Time Data Contamination
- INVITE: a Testbed of Automatically Generated Invalid Questions to Evaluate Large Language Models for Hallucinations
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- Detecting Data Errors: Where are we and what needs to be done?
- UniClean: A Scalable Data Cleaning Solution for Mixed Errors based on Unified Cleaners and Optimized Cleaning Workflow
- DataVinci: Learning Syntactic and Semantic String Repairs
- NADEEF: A Generalized Data Cleaning System
- Enhancing Hallucination Detection through Noise Injection
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Pollock: A Data Loading Benchmark