OpenRT: An Open-source Framework for Reasoning Over Tabular Data

Yilun Zhao, Boyu Mi, Zhenting Qi, Linyong Nan, Minghao Guo, Arman Cohan, Dragomir RadevACL 2023 · OpenReview

TL;DR

The paper introduces OPENRT, the first open-source framework for reasoning over tabular data, addressing the lack of standardization and compatibility in existing table pre-training methods. It provides a unified platform for reproducing, comparing, and developing new models, alongside TARAT, a tool for creating new table reasoning datasets.

What Problem Does This Paper Address?

Reasoning over tabular data, tasks like question answering, fact checking, and text generation, has seen significant advancements with table pre-training methods. However, a major hurdle is the lack of a standardized and fair evaluation environment. Existing research often benchmarks models on different datasets with varying configurations, making direct comparisons difficult and time-consuming. Furthermore, models are typically developed in isolated systems, hindering compatibility and re-implementation. This fragmentation slows down progress in the field.

Methods

OPENRT is designed with modularity, standardization, and extensibility in mind. Its core components include:

  • Configuration Module: Manages all experiment settings through command lines and configuration files, ensuring reproducible and standardized comparisons.
  • Data Module: Handles popular table reasoning datasets, converting them into a unified format. It includes a Preprocessor that tokenizes and truncates data, and TRDataset/TRDataLoader classes for managing datasets.
  • Modeling Module: Unifies implementations of various table reasoning models under a TRModel parent class, simplifying the addition of new models. It currently supports models like TAPAS, UnifiedSKG, TAPEX, REASTAP, OmniTab, and PLOG.
  • Evaluation Module: Integrates all evaluation metrics used in official implementations, allowing for straightforward performance assessment.

Additionally, the paper introduces TARAT, a Table Reasoning Annotation Tool. TARAT supports collaborative annotation for question answering, fact checking, and text generation tasks. It features quick deployment, better quality control (role isolation, progress tracking), high productivity (grammar checking, efficient fact annotation), and free accessibility.

Datasets

OPENRT includes several popular datasets for table reasoning tasks:

  • Question Answering: WIKISQL, WTQ, SQA, FETAQA
  • Fact Checking: TABFACT
  • Faithful Table-to-Text Generation: LOGICNLG

TARAT is designed to facilitate the creation of new datasets for these tasks, allowing researchers to easily integrate their custom datasets into OPENRT for evaluation.

Key Insights and Ideas

  • The Need for Standardization: The paper strongly emphasizes that the current fragmented landscape of table reasoning research hinders progress. A unified framework like OPENRT is crucial for enabling fair comparisons and accelerating development.
  • Modular Design for Extensibility: OPENRT's modular architecture is key to its utility. It allows researchers to easily swap out components, add new models, or integrate custom datasets, fostering a collaborative research ecosystem.
  • TARAT as a Dataset Enabler: The introduction of TARAT is a significant contribution, addressing the often-laborious process of dataset creation. By simplifying collaborative annotation, it aims to spur the development of new, diverse benchmarks.
  • Pre-training's Broad Applicability: The experimental results suggest that pre-training strategies, like those in PLOG and OmniTab, can significantly improve performance across various table reasoning tasks, highlighting the power of generalizable pre-training.

Key Results

The paper presents experimental results comparing several table pre-training models on FETAQA and LOGICNLG (Table 3) and on short-form QA and fact-checking datasets (Table 4).

  • Faithful Table-to-Text Generation: PLOG showed strong performance on surface-level evaluations (BLEU, ROUGE, BERTScore), attributed to its pre-training on logical forms. OmniTab excelled in faithfulness-level evaluations (SP-Acc, NLI-Acc).
  • Fact Checking and Short-Form QA: OmniTab demonstrated the best performance on most fact-checking and short-form QA tasks, indicating the effectiveness of its pre-training approach.
  • General Trend: The results underscore the benefit of pre-training models over both natural and synthetic data to enhance reasoning capabilities across different table reasoning tasks.

Limitations and Future Work

  • Focus on Pre-training: The current OPENRT framework primarily focuses on table pre-training models, acknowledging that models trained without a pre-training step are also valuable but not the primary focus.
  • Ongoing Development: The authors plan to continuously add more datasets and the latest table pre-training models to OPENRT. They actively welcome contributions from the research community to further develop and improve both OPENRT and TARAT.
  • Community Collaboration: The paper explicitly calls for researchers to join in the development and maintenance of these tools, aiming to collectively advance the field of table reasoning.

Summary updated

BibTeX
@inproceedings{zhao2023openrt,
  author = {Yilun Zhao and Boyu Mi and Zhenting Qi and Linyong Nan and Minghao Guo and Arman Cohan and Dragomir Radev},
  title = {{OpenRT: An Open-source Framework for Reasoning Over Tabular Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=RzDSxuhLy2},
}
Figure