TL;DR
This paper presents a system for fact verification over tables, a crucial task for understanding information in scientific documents. The authors tackle the challenge of automatically determining if a textual claim is supported, refuted, or unknown based on tabular data by focusing on generating logical form candidates from the table. This approach aims to bridge the gap between natural language claims and structured tabular information.
What Problem Does This Paper Address?
In today's data-rich world, tables are a common way to present information concisely, especially in scientific literature. However, verifying the truthfulness of claims made about this tabular data automatically is a significant challenge. Previous work often focused on text-based fact verification, neglecting the rich information contained within tables. This paper addresses the SemEval-2021 Task 9, which specifically targets fact verification over tables. The task involves two sub-tasks: (1) classifying a textual claim as "entailed" (supported by the table), "refuted" (contradicted by the table), or "unknown" (insufficient information in the table), and (2) identifying the specific table cells that serve as evidence for the claim. The core problem is how to effectively link natural language claims to the structured data within tables to perform accurate verification.
Methods
The system's core strategy revolves around candidate generation for logical forms derived from the table. This involves a two-step process:
- Candidate Generation: The system first parses the input claim statement and the table. It then uses a combination of keyword matching and dependency parsing to identify potential logical forms that can be applied to the table data. These logical forms represent operations that can be performed on the table to extract relevant information. For example, a claim like "The n value is the same for Hong Kong and Malaysia" might trigger a candidate logical form for comparing values across rows for a specific column.
- Candidate Execution and Verification: Once a set of candidate logical forms is generated, the system executes the most probable one. The output of this execution is then compared against the original claim statement to determine its veracity. The system also employs approximate keyword matching using the Jaro metric to align column and row labels from the table with parts of the claim statement.
The paper also details the parsing of XML documents containing tables, loading them into pandas DataFrames, and handling hierarchical column labels.
Datasets
The paper utilizes data provided by the SemEval-2021 Task 9 organizers. The data splits (train, dev, test) are described in terms of the number of documents, tables, and claims. The tables themselves are sourced from scientific articles found on ScienceDirect. The paper does not introduce a new dataset but rather uses the benchmark established for the task.
Key Insights and Ideas
- Bridging the Gap: The central idea is to translate natural language claims into executable logical forms that can operate on tabular data. This is a crucial step for enabling automated fact verification over tables.
- Keyword and Dependency Parsing Synergy: The combination of keyword matching and dependency parsing proves effective in identifying relevant table elements and potential operations for claim verification.
- Handling Hierarchical Columns: The paper acknowledges the challenge of identifying column labels with hierarchical structures and describes an approach to address this.
- Importance of Candidate Generation: The authors emphasize that the effectiveness of their system hinges on the quality and comprehensiveness of the candidate logical forms. A limited set of candidates can lead to missed verification opportunities.
Key Results
The paper reports F1 scores for Task A (Fact Verification) across train, development, and test sets.
- Task A (2-way and 3-way): The averaged F1 scores over tables are reported, with the 2-way score being identical to the 3-way score on the train data due to the absence of the "unknown" class in the ground truth. The test set F1 score for the 3-way classification is 0.3849.
- Task B (Evidence Finding): This task was not evaluated on the train data due to the absence of ground truth for relevant cell selection. The F1 score for the development set is reported as 0.4314.
- Confusion Matrices: The confusion matrices for Task A on train and test data reveal a significant number of "unknown" predictions, indicating that the system often struggles to find sufficient information or a definitive answer within the tables.
Limitations and Future Work
The authors acknowledge several limitations:
- Absence of Semantic Matching: The system can fail to match semantically similar words that are not exact matches.
- Limited Candidate Generation Rules: The current set of candidate generation rules is not exhaustive, leading to missed opportunities for verification.
- Deterministic Candidate Selection: The system selects the first matching candidate, which might not always be the most accurate.
Future work proposed includes:
- Extending Candidate Generation: Developing a more comprehensive list of candidate logical forms.
- Scoring-Based Candidate Selection: Implementing a system that scores multiple probable candidates rather than selecting the first one.
- Leveraging Feature Extraction: Incorporating ideas from prior work (like Pasupat and Liang, 2015) for feature extraction and building a log-linear model to score candidates.
Summary updated