TABARD: A Novel Benchmark for Tabular Anomaly Analysis, Reasoning and Detection

Manan Roy Choudhury, Anirudh Iyengar Kaniyar Narayana Iyengar, Shikhhar Siingh, Sugeeth Puranam, Vivek GuptaEMNLP 2025 · OpenReview

TL;DR

This paper introduces TABARD, a new benchmark for detecting fine-grained anomalies in tabular data, and evaluates Large Language Models (LLMs) on this task. They find LLMs struggle with complex reasoning and longer tables, leading them to propose novel prompting strategies that significantly improve anomaly detection accuracy and interpretability.

What Problem Does This Paper Address?

Real-world tabular data, crucial for many applications, is often riddled with subtle errors like incorrect values, logical inconsistencies, or temporal misalignments. These anomalies can cascade into downstream errors, impacting decisions. While anomaly detection is well-studied in text and images, fine-grained anomaly detection specifically within tabular data remains underexplored, lacking standardized benchmarks and systematic evaluation. This paper aims to fill this gap by creating a comprehensive benchmark and proposing methods to improve LLM performance on this challenging task.

Methods

The authors tackle tabular anomaly detection using LLMs through several innovative approaches:

  • TABARD Benchmark: They construct a novel benchmark by perturbing tables from existing datasets (WikiTQ, FeTaQA, Spider, BEAVER) to introduce eight distinct anomaly types (value, factual, logical, temporal, calculation, security, normalization, consistency). This benchmark includes paired clean and corrupted tables for supervised evaluation.
  • Prompting Strategies: They systematically evaluate LLMs using various prompting techniques, including:
  • Direct and Indirect Prompts: Ranging from simple problem statements to few-shot examples, with and without Chain-of-Thought (CoT) reasoning.
  • Multi-Reasoning & Self-Verification Prompts: Novel methods like MUSEVE (Multi-Reasoning Self Verification) and SEVCOT (Self Verification Chain of Thoughts) that leverage CoT, self-consistency, and iterative refinement.
  • Neuro-Symbolic Constraint Method (NSCM): A unique approach that converts LLM-generated constraints into executable Python code for robust validation.
  • Data Generation: They detail a systematic process for generating anomalies using LLMs, ensuring control over anomaly types and providing explanations for the perturbations.

Datasets

The TABARD benchmark is constructed by aggregating and perturbing tables from four widely-used datasets:

  • WikiTQ: Covers broad open-domain tables.
  • FeTaQA: Focuses on English free-form question answering over tables.
  • Spider: Contains complex text-to-SQL tasks across diverse domains.
  • BEAVER: Designed for fairness and bias evaluation.

Key Insights and Ideas

  • LLMs Struggle with Complex Tabular Reasoning: Standard prompting methods, especially for complex reasoning tasks and longer tables, reveal significant limitations in LLMs' ability to detect anomalies.
  • Prompt Engineering is Crucial: The effectiveness of LLMs heavily depends on the prompting strategy. Simple prompts are insufficient for nuanced anomaly detection.
  • CoT Improves Recall, but Not Always Precision: Chain-of-Thought reasoning consistently boosts recall, particularly for simpler anomaly types, but can sometimes lead to a slight decrease in precision.
  • Novel Prompting Strategies Offer Significant Gains: Advanced methods like MUSEVE, SEVCOT, and especially NSCM, which incorporates constraint-based reasoning, show substantial improvements in both precision and recall, and offer better interpretability.
  • Dataset Characteristics Matter: Table length, schema complexity, and the nature of anomalies (e.g., semantic vs. structural) significantly impact LLM performance. Shorter, more uniform tables are easier to handle.
  • Anomaly Type Interdependence: Many anomaly types frequently co-occur, highlighting the need for models that can disentangle overlapping error signals.

Key Results

  • TABARD Benchmark: The paper introduces TABARD, a benchmark with eight anomaly types, designed to evaluate LLMs.
  • Prompting Strategy Impact: Adding CoT reasoning consistently improved recall for ChatGPT-40 and Gemini-1.5-Pro, with gains ranging from 0.4% to 1.5% in recall.
  • Advanced Methods Performance:
    • MUSEVE achieved the highest precision on FeTaQA for ChatGPT-40 (48.9%).
  • SEVCOT led in precision on Spider+BEA (46.8%) and WikiTQ (56.9%) for ChatGPT-40.
  • NSCM dominated recall across datasets and models, achieving up to 66.6% recall on WikiTQ for ChatGPT-40 and 71.3% on FeTaQA for Gemini.
  • Model Performance: Gemini 1.5 Flash and GPT-40 generally outperformed LLaMA-3.1-70B-Instruct, especially on more complex datasets.
  • Dataset Variation Impact: Variation 2 (isolating anomalies to unique rows/columns) caused a significant performance drop, while Variation 3 (prioritizing harder anomalies) showed the strongest results.

Limitations and Future Work

  • Limitations:
    • The current benchmark and methods are restricted to anomalies within individual tables, not across multiple tables or inter-table reasoning.
  • The study assumes access to full, complete tables, whereas real-world data often contains missing entries.
  • The focus is on prompting strategies, not fine-tuning LLMs specifically for tabular anomaly detection.
  • The constraint-based method can have a higher false positive rate due to limited contextual information for the LLM during constraint generation.
  • Future Work:
    • Detecting mixed anomalies (multiple types in one cell) and anomalies across multiple cells.
  • Handling incomplete tables with missing data.
  • Investigating task-driven anomaly detection where anomalies are inferred from downstream task performance.
  • Improving constraint generation by incorporating feedback and meta-level modeling.
  • Extending anomaly detection to multimodal tables (text + visuals).
  • Developing approaches to address the higher false positive rate in constraint-based methods.

Summary updated

Abstract

We study the capabilities of large language models (LLMs) in detecting fine-grained anomalies in tabular data. Specifically, we examine: (1) how well LLMs can identify diverse anomaly types including factual, logical, temporal, and value-based errors; (2) the impact of prompt design and prompting strategies; and (3) the effect of table structure and anomaly type on detection accuracy. To this end, we introduce TABARD, a new benchmark constructed by perturbing tables from WikiTQ, FeTaQA, Spider, and BEAVER. The dataset spans multiple domains and eight anomaly categories, including paired clean and corrupted tables. We evaluate LLMs using direct, indirect, and Chain-of-Thought (CoT) prompting. Our results reveal notable limitations in standard prompting, especially for complex reasoning tasks and longer tables. To overcome these issues, we propose a unified framework combining multi-step prompting, self-verification, and constraint-based rule execution. Our approach significantly improves precision and recall, offering a promising direction for robust and interpretable anomaly detection in tables.
BibTeX
@inproceedings{choudhury2025tabard,
  author = {Manan Roy Choudhury and Anirudh Iyengar Kaniyar Narayana Iyengar and Shikhhar Siingh and Sugeeth Puranam and Vivek Gupta},
  title = {{TABARD: A Novel Benchmark for Tabular Anomaly Analysis, Reasoning and Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=fEJt21Iy6a},
}
Figure