Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis & Benchmark]

Jingru Yang, Honglu Zhang, Jiayi Mi, Ju Fan, Zheng Chen, Xiaoyong DuarXiv 2026 · arXiv

TL;DR

Are Large Language Models (LLMs) actually "cleaning" your data, or are they just reciting facts from their training sets? This paper introduces CoRe (Cleaning or Recalling), a benchmarking protocol that reveals how heavily LLM-based data cleaning relies on internal entity knowledge rather than generalizable logic. By swapping real-world entities for fictitious ones, the authors show that repair performance often plummets, suggesting that current benchmarks may be overestimating how well these models will perform on private, unfamiliar data.

What Problem Does This Paper Address?

As LLMs are increasingly integrated into data management pipelines for tasks like error detection and value imputation, a critical question arises: Generalization vs. Memorization.

Standard data-cleaning benchmarks (like the Hospital or Movies datasets) consist of public, well-known entities. Because LLMs are trained on massive web crawls, they likely already "know" the correct values for these records (e.g., the release year of a famous film). If a model "repairs" a record simply by recalling a fact from its parameters, it isn't actually demonstrating a capacity for data cleaning. This makes it difficult for engineers to know if a model will work on proprietary enterprise data that it has never seen before.

Methods

The authors propose the CoRe protocol, which evaluates repair performance under four distinct input conditions to isolate the source of the model's success:

  1. Original (O): The standard public benchmark data.
  2. Fresh (F): Real-world records that are newer than the model's training cutoff.
  3. Word-based Surrogate (W): Identity values (like names or IDs) are replaced one-to-one with fictitious English-word combinations.
  4. Syllable-based Surrogate (S): Identity values are replaced with meaningless, randomized syllables.

Crucially, the surrogates retain the exact same error positions and non-identity values as the original. If a model’s performance drops on the surrogates, it indicates a dependence on entity knowledge (familiarity with the specific record) rather than cleaning logic (the ability to fix patterns).

The researchers also implemented a Conflict Test, where they intentionally assigned a "wrong" but real-world-consistent value to a cell to see if the LLM would stubbornly "correct" it back to the real-world fact, ignoring the evidence provided in the table.

Datasets

The study utilized several classic data-cleaning benchmarks:

  • Hospital: Typos in hospital names and addresses.
  • Movies: Mismatched identifiers and formatting errors between IMDb and Rotten Tomatoes.
  • Beers: Formatting inconsistencies in brewery and city names.
  • Rayyan: A dataset of biomedical articles prone to spreadsheet-induced date conversions.
  • Flights & Tax: Used primarily for memorization probes.

The authors generated 21 controlled tables containing approximately 41,000 fictitious values to support the CoRe protocol.

Key Insights and Ideas

  • The "Recalling" Trap: Performance is highly sensitive to entity names. For example, on the Movies dataset, DeepSeek-V4-Pro’s accuracy in repairing film years dropped from 91% on real films to just 18% when the movie titles were replaced with fake words—even though the logic to fix the years remained identical in the text.
  • Knowledge Over Evidence: Commercial models often prioritize their internal "knowledge" over explicit instructions. In the conflict test, models frequently rewrote assigned directors to match real-world credits at rates up to 40%, even when explicitly told to treat the table as the source of truth.
  • Benchmark Quality Matters: The authors found that some benchmarks, like Rayyan, actually reward models for reproducing systematic errors (like spreadsheet date conversions) rather than providing the "correct" data. In one case, a model was penalized for providing a correct ISSN because the benchmark reference contained a corrupted date string.
  • No Row-Level Memory: Interestingly, the "continuation probe" (asking the model to predict the next row) found no evidence that models had memorized specific benchmark rows. The dependence is on entity-level facts, not the specific dataset structure.

Key Results

The study evaluated eight LLMs (including DeepSeek-V4-Pro, Qwen2.5, and GPT-6.1-Sol) and specialized cleaners like Baran and TabClean.

Figure 1: Recovery evidence in Movies. DeepSeek-V4-Pro repairs fewer identifiers when mappings disappear, but also fewer years when candidates remain. Labels give re- paired/error cells; the middle-year rule matches 167/190 ref- erences. O/W/S are paired; F uses fresh records.
  • Identity Repair: On the Hospital dataset, replacing names with syllables caused a massive drop in repair F1 (ranging from 0.59 to 0.72) for top commercial models.
  • The "Syllable" Advantage: GPT-6.1-Sol showed a unique ability to repair syllable-based names (74 out of 95 errors) by picking up on the restricted vocabulary used in the surrogate construction, whereas other models failed.
  • Commercial vs. Open-Weight: Commercial models generally showed much higher dependence on entity knowledge, while open-weight models (like Qwen2.5-7B) struggled with the cleaning task regardless of whether the entities were real or fake.

Limitations and Future Work

  • Language and Geography: The study focused exclusively on English-language tables and primarily US-based entities.
  • Model Scope: The evaluation did not include some major models like Claude or Gemini.
  • Surrogate Regularity: The authors noted that the way fake names are constructed (e.g., using a restricted set of syllables) can sometimes introduce new patterns that models might exploit, potentially confounding the results.
  • Future Direction: The authors suggest that future cleaning tools should explicitly separate "mechanical" conversions from "content" changes and that benchmarks should include "unseen" entity variants to ensure models are learning generalizable cleaning rules.

For those interested in reproducing these results or testing their own cleaners, the authors have provided their controlled tables and scoring code at https://github.com/hinsonver/CoRe-Benchmark.

Summary updated

Abstract

High scores on public data-cleaning benchmarks can reflect knowledge of familiar entities, making it difficult to assess how well an LLM repairs unfamiliar data. We present CoRE (Cleaning or Recalling), a protocol comparing public benchmarks with fresh real records and paired name surrogates built from English words or syllables. The surrogates retain error positions and non-identity values while transferring the benchmark's error mechanisms. Across 21 controlled tables and eight LLMs, repair performance varies substantially with entity names on Hospital and, for commercial models, Movies. Renaming removes some information needed for repair, such as a film's identifier mapping. It also affects repairs whose candidates remain available: replacing entity names with word-based surrogates lowers DeepSeek-V4-Pro's year accuracy from 91% to 18%. On Hospital and Movies, commercial models score higher on fresh records than on word-based surrogates. A screened continuation probe finds no positive evidence of benchmark-row memory in the tested models. Familiar facts can conflict with the task. Even with explicit table-evidence instructions, commercial models rewrite assigned directors at rates of 0.000-0.404 on known films. Reference scoring can also disagree with external records. On Rayyan, an ISSN correction matching a title-matched PubMed record receives no credit, whereas 94% of TabClean's credited repairs reproduce the reference date transformation. We provide controlled tables, scoring code and materials for offline replay.
BibTeX
@misc{yang2026cleaning,
  author = {Jingru Yang and Honglu Zhang and Jiayi Mi and Ju Fan and Zheng Chen and Xiaoyong Du},
  title = {{Cleaning or Recalling? Knowledge Dependence in LLM-based Data Cleaning [Experiment, Analysis \& Benchmark]}},
  year = {2026},
  eprint = {2611.08106},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08106},
}
Figure