2Columns1Row: A Russian Benchmark for Textual and Multimodal Table Understanding and Reasoning

Vildan Saburov, Daniil Vodolazsky, Danil Sazanakov, Alena Fenogenova2025 · DOI

TL;DR

This paper introduces 2Columns1Row, the first comprehensive benchmark for table understanding and reasoning in Russian, supporting both textual and multimodal inputs. It addresses the critical need for robust evaluation of Large Language Models (LLMs) on complex tabular data in non-English languages, revealing current model limitations and paving the way for improved AI capabilities in document analysis.

What Problem Does This Paper Address?

Automated understanding of tabular data is a crucial but challenging task in document processing, often referred to as Table Question Answering (TableQA). While LLMs have made significant strides in Natural Language Processing, their performance on complex table-related tasks, especially in languages other than English, remains a bottleneck. Existing benchmarks are often limited in complexity or language coverage, hindering the development and evaluation of models for real-world applications. This paper aims to bridge this gap by introducing a new, challenging benchmark specifically designed for Russian tabular data, encompassing both text-only and multimodal inputs.

Methods

The core of this work is the 2Columns1Row benchmark, a synthetically generated dataset designed to test LLMs' ability to retrieve a value from one column based on a value in another. The benchmark comprises six diverse datasets, totaling 28,800 tables, which vary in complexity based on cell content uniformity and textual richness. For multimodal evaluation, full-size screenshots of tables were generated. The authors also explored the impact of different prompt engineering techniques, comparing "usual" and "refined" system prompts to guide LLMs. They investigated various table text representations (Markdown, JSON, CSV, HTML) to determine the most effective format for LLM input. Finally, they conducted experiments with both text-only LLMs and multimodal Large Vision-Language Models (LVLMs), including an analysis of supervised fine-tuning (SFT) on a subset of models.

Datasets

The paper introduces the 2Columns1Row benchmark, a novel, synthetically generated dataset for Russian TableQA. It consists of six distinct datasets:

  • Person Info: Contains personal details like names, addresses, and phone numbers.
  • Person Info Hard: An advanced version with more complex data structures and synthetic word sequences.
  • Colors: Features color values in hexadecimal format (#RRGGBB).
  • Numbers: Includes floating-point numbers with six decimal places.
  • Company Info: Contains company names, addresses, fax numbers, and other related information.
  • Word Sequences: Comprises Russian words, their combinations, and sentence fragments.

These datasets are designed to vary in complexity, cell value uniformity, and textual content, providing a robust evaluation suite.

Key Insights and Ideas

  • The Russian language barrier: The lack of comprehensive benchmarks for non-English languages like Russian significantly hinders progress in developing and evaluating LLMs for table understanding in diverse global contexts.
  • Complexity matters: Even advanced LLMs struggle with tables that have high textual complexity or uniform values, suggesting that current models are not yet adept at nuanced reasoning across diverse tabular structures.
  • Multimodal models still lag: Despite combining visual and textual information, LVLMs generally underperform compared to their text-only counterparts on this specific task, highlighting challenges in integrating visual and textual understanding for tabular data.
  • Prompt engineering is crucial: The choice of system prompt can significantly impact LLM performance, with more detailed and structured prompts leading to better results, especially for models like Llama.
  • Dynamic evaluation is key: The benchmark's design allows for dynamic regeneration, ensuring its integrity and mitigating potential data contamination issues, which is vital for reliable model evaluation.

Key Results

The paper evaluated over 25 LLMs and LVLMs on the 2Columns1Row benchmark. Key findings include:

  • GPT-40 demonstrated exceptionally strong performance across all datasets.
  • Qwen models showed excellent results, with Qwen-2.5-32B-Instruct even outperforming the larger Qwen-2.5-72B-Instruct model on some metrics.
  • Text-only LLMs generally outperformed LVLMs, with the exception of Qwen-2.5-VL-72B-Instruct, which showed competitive results.
  • TableLLM-8B was an outlier, performing poorly across most datasets.
  • Heatmap analysis revealed that model performance deteriorates with an increasing number of columns and struggles with rows further from the table's beginning, indicating challenges with large tables.
  • Multimodal models showed a lower ratio of "false cells" to "non-existent values" compared to text-only models, suggesting a greater propensity for hallucinations in LVLMs.

Limitations and Future Work

The authors acknowledge several limitations and outline future directions:

  • Task Scope and Complexity: The current benchmark focuses on row/column relationship understanding. Future work will expand to more complex tasks like table summarization, multi-step reasoning, and integration with autonomous AI agents.
  • Real-World Data: The benchmark relies on synthetically generated data. Future work aims to incorporate complex, real-world datasets with features like multi-level headers and merged cells to better reflect practical challenges.
  • Algorithmic Question Generation: While ensuring consistency, the algorithmic generation of questions may limit linguistic diversity. Future work will explore more naturalistic query generation.
  • Dynamic Structure: Developing a more dynamic benchmark structure is crucial for mitigating data contamination and leakage issues in future evaluations.

Summary updated

BibTeX
@misc{saburov2025columns,
  author = {Vildan Saburov and Daniil Vodolazsky and Danil Sazanakov and Alena Fenogenova},
  title = {{2Columns1Row: A Russian Benchmark for Textual and Multimodal Table Understanding and Reasoning}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.721},
}
Figure