Quality Assessment of Tabular Data using Large Language Models and Code Generation

Ashlesha Akella, Akshar Kaul, Krishnasuri Narayanam, Sameep MehtaEMNLP 2025

TL;DR

This paper introduces a novel framework that leverages Large Language Models (LLMs) to automatically generate data quality rules and executable code for tabular data. By combining statistical inlier detection with LLM-driven rule generation and code synthesis, the approach significantly outperforms existing methods in identifying data quality issues, making data cleaning more efficient and accessible.

What Problem Does This Paper Address?

Ensuring the quality of tabular data is paramount for reliable analysis and decision-making. However, traditional data quality validation methods, such as rule-based systems, often struggle with inefficiency, require significant human intervention, and are computationally expensive, especially for large, high-dimensional datasets. While statistical methods can detect structural anomalies, they often miss context-dependent errors that rely on semantic understanding or external knowledge. Existing LLM-based solutions, while promising, can be computationally prohibitive due to per-row inference or dataset-specific training. This paper addresses the need for an automated, scalable, and semantically aware approach to data quality assessment that can effectively identify complex errors without excessive computational cost.

Methods

The proposed framework operates in three stages:

  1. Inlier-Outlier Detection: The system first preprocesses the raw tabular data to identify and filter out "outlier" rows using a distributed clustering technique and the Sparx algorithm. This step ensures that subsequent analysis is performed on a cleaner subset of the data, reducing noise and improving the accuracy of rule generation.
  2. LLM-Driven Rule Generation and Enrichment: For the identified inlier data, LLMs (specifically Gemma-3-12B) are prompted to generate data quality rules. This process is enhanced by:
  3. Retrieval-Augmented Generation (RAG): Leveraging external knowledge bases and domain-specific few-shot examples to guide the LLM.
  4. Rule Enrichment: Adding detailed specifications and pseudocode to the generated rules to make them more precise and executable.
  5. Multi-Layer Guardrails: Implementing filters to resolve conflicts between rules, identify redundancies, and evaluate rules based on a rubric to ensure their value and correctness.
  6. Python Code Generation: A specialized LLM (Qwen-2.5-Coder) translates the validated rule cards into executable Python code. This allows for fine-grained, cell-level validation by generating a Boolean error mask when applied to the dataset.

Datasets

The authors evaluated their framework on several benchmark datasets, including:

  • Clean Benchmark Datasets: Adult, Beers, Bikes, Breast Cancer, HAR, Movies, Nasa, Rayyan, Soil Moisture, and Tax.
  • Synthetic Error Injection: They used the REIN corruption model to inject various types of errors (typos, missing values, cell swaps, Gaussian noise) at different noise levels (10%, 20%, 30%) into these datasets.
  • Domain-Specific Datasets for Robustness: Datasets covering manufacturing (3D Printer), environmental monitoring (Water-Quality), scholarly communication (Citation), and energy consumption (Power).

Key Insights and Ideas

  • LLMs as Semantic Rule Generators: The core idea is that LLMs, with their understanding of language and context, can go beyond simple statistical checks to identify semantically meaningful data quality issues that traditional methods miss.
  • The Power of Few-Shot Learning: Domain-specific few-shot examples are crucial for steering LLMs to generate relevant and accurate rules, significantly improving both confidence and coverage.
  • Guardrails are Essential: Simply generating rules with an LLM is not enough. Multi-layered guardrails (conflict resolution, rubric evaluation) are vital for ensuring the quality, consistency, and usefulness of the generated rules.
  • Inlier Detection Boosts Accuracy: Filtering out noisy data before rule generation is critical. The inlier detection module significantly improves the accuracy of the LLM's rule generation and subsequent code synthesis.
  • Code Generation for Actionability: Translating rules into executable code makes the data quality assessment actionable, enabling direct application to datasets.

Key Results

  • Superior Accuracy: The proposed framework consistently outperformed existing error detection techniques, achieving the highest F₁ scores in 26 out of 30 settings across ten datasets and three noise levels.
  • Significant Improvement on Challenging Tables: The method showed marked superiority on difficult datasets like Bikes, Breast Cancer, and Soil-Moisture.
  • Domain-Specific Prompts Matter: Using domain-tailored few-shot examples led to a notable increase in F₁ scores, demonstrating the importance of contextualizing LLM prompts.
  • Reduced Hallucinations: The use of few-shot examples and guardrails helped contain LLM hallucinations and improve rule reliability.

Limitations and Future Work

  • Single-Table Setting: The current framework is designed for single tabular datasets. The authors acknowledge that real-world data often involves multiple related tables, and extending the framework to handle inter-table dependencies is a key area for future work.
  • Extensibility: The framework is designed to be extensible, allowing for the addition of new DQ rule types by simply providing corresponding prompt templates and example rule cards.
  • SQL Generation: While Python code generation was demonstrated, the framework also supports source code generation in SQL, which can be beneficial for database environments.

Summary updated

BibTeX
@inproceedings{akella2025quality,
  author = {Ashlesha Akella and Akshar Kaul and Krishnasuri Narayanam and Sameep Mehta},
  title = {{Quality Assessment of Tabular Data using Large Language Models and Code Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure