Abstract
Large Language Models (LLMs) have demonstrated significant capabilities in text processing, yet their proficiency in direct structural manipulation of tabular data remains a challenge. We introduce WikiTableEdit, a comprehensive benchmark designed to evaluate LLMs on their ability to edit tables using natural language instructions. Spanning over 200,000 instances and six core editing operations, the benchmark moves beyond code-generation paradigms toward direct HTML-to-HTML transformations. Our analysis identifies a significant "irregularity tax"—a sharp decline in performance when models encounter tables with merged cells—and underscores the importance of structural awareness in data-centric AI tasks.
Introduction
Tables are a cornerstone of human information organization, ranging from simple grids to complex "irregular" structures with merged cells. While LLMs are frequently used for table-to-text or text-to-SQL tasks, the ability to directly modify a table's structure via natural language (e.g., "Swap the 1st and 4th rows") is a more intuitive but technically demanding frontier. Existing systems often rely on intermediate programming languages, which can be brittle and inaccessible to non-programmers. WikiTableEdit addresses this by tasking models with direct structural editing, revealing that LLMs still struggle with the underlying "geometry" of tabular data.
The WikiTableEdit Dataset
The WikiTableEdit dataset is constructed from 26,531 unique Wikipedia tables, expanding upon the WikiSQL foundation. It includes 194,996 training instances and 28,706 test instances. The benchmark covers six fundamental operations: adding/removing rows or columns, swapping rows, reordering (sorting) based on a column, and merging/splitting adjacent cells.
These operations range from simple deletions to complex "composite" tasks like reordering, which requires the model to maintain perfect indexing across multiple simultaneous changes.
Methodology: HTML and the TED Metric
To facilitate structural reasoning, tables are represented in HTML format, allowing models to process attributes like rowspan and colspan. However, traditional text metrics like BLEU or ROUGE are insufficient for tables; a single misplaced tag can invalidate the entire structure regardless of textual accuracy.
To solve this, the researchers introduced Table Edit Distance (TED). This metric evaluates two distinct components:
- Content Distance: Measures changes to the text within cells.
- Structural Distance: Measures changes to the table's "geometry" by comparing Rowspan Matrices.
By representing the table as a matrix of span values, the TED metric can precisely quantify how well a model preserved or correctly altered the table's layout.
Key Results and the "Irregularity Tax"
Experiments involving models such as LLaMA2-7B, ChatGLM3-6B, and GPT-3.5-turbo revealed several critical insights:
- The Difficulty of Irregularity: Editing "irregular" tables (those with merged cells) is significantly more difficult. For instance, GPT-3.5-turbo's performance plummeted by nearly 90% when transitioning from regular to irregular table tasks. Swapping rows in an irregular table requires recalculating spans for cells that were not even part of the swap.
- Zero-Shot vs. Fine-Tuning: In zero-shot settings, even GPT-3.5-turbo achieved an Exact Match (EM) score of only 6.34%. However, supervised fine-tuning (SFT) provided a substantial boost, with fine-tuned LLaMA2-7B and ChatGLM3-6B models reaching approximately 25% EM.
- Operation Complexity: Simple additions and deletions were handled relatively well. In contrast, swapping and reordering proved the most challenging, as they necessitate complex internal indexing and structural re-mapping.
Conclusion
WikiTableEdit highlights a significant gap in the structural reasoning capabilities of current LLMs. While fine-tuning improves performance, the "irregularity tax" suggests that models do not yet fully grasp the spatial relationships inherent in tabular data. Future research should focus on specialized pre-training or architectural modifications that enable models to better internalize the diverse and often complex forms of real-world data structures.
Summary updated