WikiTableEdit: A Benchmark for Table Editing by Natural Language Instruction

Zheng Li, Xiang Chen, Xiaojun WanarXiv 2024 · DOI · arXiv

Abstract

Large Language Models (LLMs) have demonstrated significant capabilities in text processing, yet their proficiency in direct structural manipulation of tabular data remains a challenge. We introduce WikiTableEdit, a comprehensive benchmark designed to evaluate LLMs on their ability to edit tables using natural language instructions. Spanning over 200,000 instances and six core editing operations, the benchmark moves beyond code-generation paradigms toward direct HTML-to-HTML transformations. Our analysis identifies a significant "irregularity tax"—a sharp decline in performance when models encounter tables with merged cells—and underscores the importance of structural awareness in data-centric AI tasks.

Introduction

Tables are a cornerstone of human information organization, ranging from simple grids to complex "irregular" structures with merged cells. While LLMs are frequently used for table-to-text or text-to-SQL tasks, the ability to directly modify a table's structure via natural language (e.g., "Swap the 1st and 4th rows") is a more intuitive but technically demanding frontier. Existing systems often rely on intermediate programming languages, which can be brittle and inaccessible to non-programmers. WikiTableEdit addresses this by tasking models with direct structural editing, revealing that LLMs still struggle with the underlying "geometry" of tabular data.

The WikiTableEdit Dataset

The WikiTableEdit dataset is constructed from 26,531 unique Wikipedia tables, expanding upon the WikiSQL foundation. It includes 194,996 training instances and 28,706 test instances. The benchmark covers six fundamental operations: adding/removing rows or columns, swapping rows, reordering (sorting) based on a column, and merging/splitting adjacent cells.

Examples of core table editing operations in the WikiTableEdit benchmark: (a) removing a row, (b) swapping rows, and (c) reordering based on a column.

These operations range from simple deletions to complex "composite" tasks like reordering, which requires the model to maintain perfect indexing across multiple simultaneous changes.

Methodology: HTML and the TED Metric

To facilitate structural reasoning, tables are represented in HTML format, allowing models to process attributes like rowspan and colspan. However, traditional text metrics like BLEU or ROUGE are insufficient for tables; a single misplaced tag can invalidate the entire structure regardless of textual accuracy.

To solve this, the researchers introduced Table Edit Distance (TED). This metric evaluates two distinct components:

  1. Content Distance: Measures changes to the text within cells.
  2. Structural Distance: Measures changes to the table's "geometry" by comparing Rowspan Matrices.
The conversion of an input table into a Rowspan Matrix, a key step in calculating the Structural Distance component of the TED metric.

By representing the table as a matrix of span values, the TED metric can precisely quantify how well a model preserved or correctly altered the table's layout.

Key Results and the "Irregularity Tax"

Experiments involving models such as LLaMA2-7B, ChatGLM3-6B, and GPT-3.5-turbo revealed several critical insights:

  • The Difficulty of Irregularity: Editing "irregular" tables (those with merged cells) is significantly more difficult. For instance, GPT-3.5-turbo's performance plummeted by nearly 90% when transitioning from regular to irregular table tasks. Swapping rows in an irregular table requires recalculating spans for cells that were not even part of the swap.
  • Zero-Shot vs. Fine-Tuning: In zero-shot settings, even GPT-3.5-turbo achieved an Exact Match (EM) score of only 6.34%. However, supervised fine-tuning (SFT) provided a substantial boost, with fine-tuned LLaMA2-7B and ChatGLM3-6B models reaching approximately 25% EM.
  • Operation Complexity: Simple additions and deletions were handled relatively well. In contrast, swapping and reordering proved the most challenging, as they necessitate complex internal indexing and structural re-mapping.

Conclusion

WikiTableEdit highlights a significant gap in the structural reasoning capabilities of current LLMs. While fine-tuning improves performance, the "irregularity tax" suggests that models do not yet fully grasp the spatial relationships inherent in tabular data. Future research should focus on specialized pre-training or architectural modifications that enable models to better internalize the diverse and often complex forms of real-world data structures.

Summary updated

Abstract

Tabular data, as a crucial form of data representation, exists in diverse formats on the Web. When confronted with complex and irregular tables, manual modification becomes a laborious task. This paper investigates the performance of Large Language Models (LLMs) in the context of table editing tasks. Existing research mainly focuses on regular-shaped tables, wherein instructions are used to generate code in SQL, Python, or Excel Office-script for manipulating the tables. Nevertheless, editing tables with irregular structures, particularly those containing merged cells spanning multiple rows, poses a challenge when using code. To address this, we introduce the WikiTableEdit dataset. Leveraging 26,531 tables from the WikiSQL dataset, we automatically generate natural language instructions for six distinct basic operations and the corresponding outcomes, resulting in over 200,000 instances. Subsequently, we evaluate several representative large language models on the WikiTableEdit dataset to demonstrate the challenge of this task. The dataset will be released to the community to promote related researches.
BibTeX
@misc{li2024wikitableedit,
  author = {Zheng Li and Xiang Chen and Xiaojun Wan},
  title = {{WikiTableEdit: A Benchmark for Table Editing by Natural Language Instruction}},
  year = {2024},
  eprint = {2403.02962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.02962},
}
Figure