TL;DR
Are Large Language Models (LLMs) actually "cleaning" your data, or are they just reciting facts from their training sets? This paper introduces CoRe (Cleaning or Recalling), a benchmarking protocol that reveals how heavily LLM-based data cleaning relies on internal entity knowledge rather than generalizable logic. By swapping real-world entities for fictitious ones, the authors show that repair performance often plummets, suggesting that current benchmarks may be overestimating how well these models will perform on private, unfamiliar data.
What Problem Does This Paper Address?
As LLMs are increasingly integrated into data management pipelines for tasks like error detection and value imputation, a critical question arises: Generalization vs. Memorization.
Standard data-cleaning benchmarks (like the Hospital or Movies datasets) consist of public, well-known entities. Because LLMs are trained on massive web crawls, they likely already "know" the correct values for these records (e.g., the release year of a famous film). If a model "repairs" a record simply by recalling a fact from its parameters, it isn't actually demonstrating a capacity for data cleaning. This makes it difficult for engineers to know if a model will work on proprietary enterprise data that it has never seen before.
Methods
The authors propose the CoRe protocol, which evaluates repair performance under four distinct input conditions to isolate the source of the model's success:
- Original (O): The standard public benchmark data.
- Fresh (F): Real-world records that are newer than the model's training cutoff.
- Word-based Surrogate (W): Identity values (like names or IDs) are replaced one-to-one with fictitious English-word combinations.
- Syllable-based Surrogate (S): Identity values are replaced with meaningless, randomized syllables.
Crucially, the surrogates retain the exact same error positions and non-identity values as the original. If a model’s performance drops on the surrogates, it indicates a dependence on entity knowledge (familiarity with the specific record) rather than cleaning logic (the ability to fix patterns).
The researchers also implemented a Conflict Test, where they intentionally assigned a "wrong" but real-world-consistent value to a cell to see if the LLM would stubbornly "correct" it back to the real-world fact, ignoring the evidence provided in the table.
Datasets
The study utilized several classic data-cleaning benchmarks:
- Hospital: Typos in hospital names and addresses.
- Movies: Mismatched identifiers and formatting errors between IMDb and Rotten Tomatoes.
- Beers: Formatting inconsistencies in brewery and city names.
- Rayyan: A dataset of biomedical articles prone to spreadsheet-induced date conversions.
- Flights & Tax: Used primarily for memorization probes.
The authors generated 21 controlled tables containing approximately 41,000 fictitious values to support the CoRe protocol.
Key Insights and Ideas
- The "Recalling" Trap: Performance is highly sensitive to entity names. For example, on the Movies dataset, DeepSeek-V4-Pro’s accuracy in repairing film years dropped from 91% on real films to just 18% when the movie titles were replaced with fake words—even though the logic to fix the years remained identical in the text.
- Knowledge Over Evidence: Commercial models often prioritize their internal "knowledge" over explicit instructions. In the conflict test, models frequently rewrote assigned directors to match real-world credits at rates up to 40%, even when explicitly told to treat the table as the source of truth.
- Benchmark Quality Matters: The authors found that some benchmarks, like Rayyan, actually reward models for reproducing systematic errors (like spreadsheet date conversions) rather than providing the "correct" data. In one case, a model was penalized for providing a correct ISSN because the benchmark reference contained a corrupted date string.
- No Row-Level Memory: Interestingly, the "continuation probe" (asking the model to predict the next row) found no evidence that models had memorized specific benchmark rows. The dependence is on entity-level facts, not the specific dataset structure.
Key Results
The study evaluated eight LLMs (including DeepSeek-V4-Pro, Qwen2.5, and GPT-6.1-Sol) and specialized cleaners like Baran and TabClean.
- Identity Repair: On the Hospital dataset, replacing names with syllables caused a massive drop in repair F1 (ranging from 0.59 to 0.72) for top commercial models.
- The "Syllable" Advantage: GPT-6.1-Sol showed a unique ability to repair syllable-based names (74 out of 95 errors) by picking up on the restricted vocabulary used in the surrogate construction, whereas other models failed.
- Commercial vs. Open-Weight: Commercial models generally showed much higher dependence on entity knowledge, while open-weight models (like Qwen2.5-7B) struggled with the cleaning task regardless of whether the entities were real or fake.
Limitations and Future Work
- Language and Geography: The study focused exclusively on English-language tables and primarily US-based entities.
- Model Scope: The evaluation did not include some major models like Claude or Gemini.
- Surrogate Regularity: The authors noted that the way fake names are constructed (e.g., using a restricted set of syllables) can sometimes introduce new patterns that models might exploit, potentially confounding the results.
- Future Direction: The authors suggest that future cleaning tools should explicitly separate "mechanical" conversions from "content" changes and that benchmarks should include "unseen" entity variants to ensure models are learning generalizable cleaning rules.
For those interested in reproducing these results or testing their own cleaners, the authors have provided their controlled tables and scoring code at https://github.com/hinsonver/CoRe-Benchmark.
Summary updated