Data cleaning systems are usually evaluated by corrupting a clean table with synthetic errors and counting how many corrupted cells each system restores exactly. This scoring quietly assumes that every injected error has one recoverable answer. Often it does not. If a hospital record reads "Birmingham, AL 3523x", several original zip codes may be equally consistent with the rest of the data. The correct output in that case is a calibrated spread of belief over the possible repairs, not a single guess. A newer class of cleaners produces exactly that kind of output, a probability for each candidate repair, and the uncertain-database literature already knows how to push such probabilities through downstream queries. The benchmarks have not caught up. They reward lucky, overconfident guesses on ambiguous errors and give no credit for honest uncertainty.
We propose an error injector that controls how ambiguous each injected error is, and computes the exact probability distribution over clean tables implied by each corruption. Cleaners can then be scored with proper probabilistic metrics, both cell by cell and on the answers to downstream queries. The central question is whether ambiguity-controlled synthetic benchmarks rank cleaners the way real dirty data does, where today's uniform typo injection does not. This is the data-cleaning version of a failure already documented for language models: synthetic errors train and evaluate the wrong thing.
Core Claim
Our claim is that the ambiguity of injected errors is a hidden variable that decides which cleaners look good, and that current injection practice sets it arbitrarily. Two things follow from making it explicit. First, at low ambiguity, cleaners that output a single repair and cleaners that output a probability for each repair should be indistinguishable. As ambiguity rises, the probabilistic ones should pull ahead under proper scoring, while plain cell-level F1 hides or reverses the gap. Second, cleaner rankings on real dirty tables should be predicted better by a synthetic benchmark whose ambiguity profile is matched to the real data than by one built from uniform random corruption.
Where This Sits
Two lines of prior work supply the pieces. On the injection side, Messing Up with BART turns integrity rules into queries that select cells to corrupt. It introduces repairability, a measure of how uniquely a cleaner could infer the original value, and shows that higher repairability goes with higher repair accuracy. BART supplies our method: we reuse its notion of repairability. Its scope is narrower than ours, because it modifies deterministic values and evaluates point repairs, and it never produces a ground-truth distribution over repairs. Repairability for BART is a diagnostic. In our injector it is the quantity being controlled.
A probabilistic database approach to autoencoder-based data cleaning (DCAE) treats cells as probability distributions and injects noise into tables generated from Bayesian networks with known dependencies. It reports more than 95% F1 at detecting corrupted categorical cells but only about 25% removal of categorical noise. We read that gap as a sign that many of its errors may simply be unrecoverable as point repairs. DCAE motivates our generator design. Its evaluation still scores point recovery, and it cannot produce confidence intervals.
On the uncertainty side, PClean cleans data by Bayesian inference over an explicit model of clean data plus an error process. It reports calibrated posterior probabilities per cell, along with F1 0.90 on Flights against HoloClean's 0.41–0.64. PClean is the system that the current benchmarks under-credit, and it is our strongest competitor. The query-level machinery comes from Database Theory in Action, whose ProvSQL system records how each query answer was derived from input tuples and then computes answer probabilities from that record. It also comes from Efficient Uncertainty Tracking for Complex Queries with Attribute-level Bounds, whose AU-DBs carry lower and upper bounds through aggregation in polynomial time. Probabilistic Web Data Management and Learning from Uncertain Data motivate the work: they argue that collapsing uncertainty early to one "truth" produces wrong or arbitrary downstream results.
The cautionary result that shapes our second claim is Synthetic Error Injection Fails to Elicit Self-Correction In Language Models. It shows models detecting injected Sudoku errors at 94% but their own natural errors at 8%. We take seriously the possibility that synthetic dirt misleads in the same way. That is why the decisive test below is transfer to real dirty data rather than performance on our own benchmark. The text-to-SQL audits (Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards, which found 52.8% annotation errors on BIRD Mini-Dev and rank swings of up to 9 positions) are a reminder that real "ground truth" is itself noisy. We therefore treat real-data rankings as evidence, not as an oracle.
Research Program
The injector works over clean tables drawn from a known generative model: a Bayesian network over attributes, either specified by hand or fit to a real clean table such as Hospital. We pair this model with a parameterized error channel covering typos, swaps between values in the same domain, and value substitutions that break the table's integrity rules. Because both the clean-data model and the channel are known, we can compute the exact posterior over each corrupted cell's original value given the dirty row. We call this the ground-truth posterior.
For a candidate corruption of cell $c$ into value $d$, the injector computes the entropy of the ground-truth posterior,
$$H(c \mid d) = -\sum_{v} p(v \mid d, \text{row}) \log p(v \mid d, \text{row}),$$
where $v$ ranges over possible original values. It then accepts or rejects corruptions so that the injected set hits a target entropy profile at a fixed overall error rate. Low entropy corresponds to BART's highly repairable errors, and high entropy to errors where several originals are equally plausible.
For small domains, exact inference by enumeration is feasible. For larger ones we restrict the parts of the network that matter to bounded treewidth, so exact inference stays cheap. This mirrors the tractability conditions discussed in Database Theory in Action.
Scoring happens at two levels. At the cell level we use log-score and expected calibration error against the ground-truth posterior, alongside conventional F1. At the query level we load each cleaner's probabilistic output into ProvSQL, or into an AU-DB for aggregates. We then measure whether the cleaner's 90% credible answer set for selection, join and SUM/COUNT queries contains the answer under the true clean table, and how wide that set is.
A cleaner that outputs only point repairs is entered as a database with probability 1 on each repair. This is the fair reading of what it claims. The query level matters because errors in different cells are correlated: a cleaner can be calibrated cell by cell and still wrong about aggregates.
Finally, we estimate the ambiguity profile of real dirty tables, namely Flights, Rents and a manually verified sample of Medicare. We do this by fitting the generative model plus error channel and computing posterior entropies on the cells that are known to be erroneous. We then build matched synthetic benchmarks with the same profile.
Baselines
The naive reference point has two forms: leave the dirty value in place, or replace a flagged cell with the column mode. These set the floor every cleaner must clear at each ambiguity level, and they show how much of a score comes merely from detecting errors.
The current-practice baseline is the evaluation protocol rather than a cleaner. It uses uniform random corruption at the same overall error rate, as on the Hospital benchmark and in BART's default configuration, scored with cell-level F1. The ambiguity-targeted injector must beat this protocol at predicting real-data rankings, using the same set of cleaners, the same error rates and the same clean source tables.
The strongest existing cleaner is PClean, which reports F1 0.90 on Flights. We first reproduce that number to confirm our setup. PClean is the right adversary because it already outputs calibrated posteriors. If ambiguity-controlled scoring merely confirms that PClean wins everywhere, our benchmark adds little beyond its paper. For this reason we also include HoloClean-style point cleaners, DCAE (A probabilistic database approach to autoencoder-based data cleaning), and several deliberately miscalibrated PClean variants, giving roughly fifteen systems so that ranking correlations have some statistical power. Every system gets the same rows, the same constraints and the same tuning budget, measured in wall-clock inference time, on the same hardware.
The key ablation holds the overall error rate and error types fixed and varies only the target entropy profile. If cleaner rankings do not move, ambiguity is not the hidden variable we claim. A second ablation scores identical cleaner outputs at the cell level only versus the query level, to test whether query-level evaluation adds information.
Why not just…?
Why not just add expected calibration error to BART-style benchmarks and stop at the cell level? This is the most serious threat. If cell-level calibration ranks cleaners the same way query-level coverage does, the provenance and AU-DB machinery is unnecessary for evaluation. We will test it directly by computing the rank correlation between the two scorings on identical outputs. Our hypothesis is that they diverge on aggregate queries over correlated repairs, as in a functional dependency where one wrong city propagates. If they do not diverge, we will report that and drop query-level scoring.
Why not just evaluate on real dirty data like Flights? Real benchmarks are few, offer no control over ambiguity, and have ground truth that is itself noisy, as the text-to-SQL audits show for other domains. Real data is still the arbiter of our transfer claim, so we use it as the target, not as the instrument.
Why not just estimate query-answer distributions by Monte Carlo sampling instead of provenance circuits? For evaluation this is legitimate. We will use sampling as a reference implementation and check that coverage estimates agree. The AU-DB paper reports up to a 570% speedup over the Monte Carlo system MCDB on aggregation, so the circuit and bounds backends matter only for scaling to Medicare-sized tables, not for the scientific claim.
Decisive Checks
Two outcomes would refute the proposal. The first is that rankings under the ambiguity-matched synthetic benchmark correlate no better with rankings on Flights, Rents and the Medicare sample than rankings under uniform injection. The second is that the gap between probabilistic and point cleaners fails to widen with injected entropy on the Bayesian-network benchmarks. A positive result would look like this: at low entropy, the point and probabilistic cleaners tie on log-score, while at high entropy PClean-type systems lead on log-score and query coverage even where cell F1 shows them tied or behind. A third, diagnostic check compares the entropy profile estimated from real errors with BART's uniform profile. If they already match, the injector's value is in the scoring, not the generation, and we will say so.
References
- Messing Up with BART: Error Generation for Evaluating Data-Cleaning Algorithms
- Autoencoder-based cleaning in probabilistic databases
- PClean: Bayesian Data Cleaning at Scale with Domain-Specific Probabilistic Programming
- Database Theory in Action: Making Provenance and Probabilistic Database Theory Work in Practice (Invited Talk)
- Efficient Uncertainty Tracking for Complex Queries with Attribute-level Bounds
- Probabilistic Web Data Management
- Learning from Uncertain Data: From Possible Worlds to Possible Models
- Synthetic Error Injection Fails to Elicit Self-Correction In Language Models
- Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards