Abstract: Xiaoyu Yang, Feng Nie, Yufei Feng, Quan Liu, Zhigang Chen, Xiaodan Zhu. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).
2020.
Introduction
Fact verification—the task of determining whether a statement is supported or refuted by evidence—has seen significant progress in the domain of unstructured text. However, verifying statements against structured tables presents a unique set of challenges. Unlike plain text, tables require a combination of linguistic understanding and symbolic reasoning. For instance, a statement like "Ji-young Oh played more tournaments in 2008 than any other year" requires identifying the correct row, comparing values across the entire column, and understanding the concept of a maximum value.
This paper addresses two primary hurdles in table-based fact-checking:
- Symbolic Reasoning: Standard language models like BERT are not inherently designed to perform mathematical operations such as
max,count, oraverage. - The Semantic Gap: There is a fundamental disconnect between rigid logical programs (the symbolic representation of a query) and the fluid natural language of the statement.
To bridge this gap, the authors introduce ProgVGAT (Program-enhanced Verbalization and Graph ATtention Network), a framework that translates symbolic logic into natural language "verbalizations" and fuses them with table data using a graph-based architecture.
The ProgVGAT Framework
The ProgVGAT framework operates in three distinct stages to ensure robust reasoning and accurate verification.
1. Program Selection with Margin Loss
Because a single statement can be mapped to multiple logical programs, the model must select the most faithful representation. The authors employ a BERT-based selector trained with a novel margin loss. This strategy helps the model distinguish between truly correct programs and "spurious" ones—programs that might accidentally yield the correct True/False label through flawed logic.
2. Verbalization with Program Execution
This stage is the core innovation of the paper. Rather than feeding raw symbolic code to the model, ProgVGAT executes the program step-by-step. Each intermediate result is converted into a natural language sentence using pre-defined templates. For example, the operation eq(max(played), 29) is verbalized as "The maximum value of the column 'played' is 29." This allows the model to treat logical evidence as standard text, leveraging the pre-trained linguistic power of BERT.
3. Graph-based Verification
To synthesize the evidence, the model constructs a heterogeneous graph. This graph includes nodes for the verbalized program steps (Prog-Exec Nodes), specific table cells (Entity Nodes), and a global representation of the table and statement (Table-BERT Node). A Graph Attention Network (GAT) facilitates information exchange between these nodes, followed by a gated attention mechanism to produce the final classification.
Experimental Results
The model was evaluated on the TABFACT dataset, a large-scale benchmark containing over 118,000 statement-table pairs. ProgVGAT achieved a new state-of-the-art accuracy of 74.4% on the test set, outperforming previous models like LogicalFactChecker (71.7%).
Key findings from the results include:
- Complex Reasoning: On the "complex" subset of TABFACT, which requires multi-step logic, ProgVGAT reached 67.6% accuracy, a significant improvement over the 58.2% achieved by the Table-BERT baseline.
- Ablation Insights: Removing the Graph Attention component resulted in a performance drop, confirming that the structural fusion of symbolic and linguistic evidence is vital.
- Human-AI Gap: Despite these gains, a substantial gap remains between the model (74.4%) and human performance (92.1%), particularly on complex reasoning tasks.
Conclusion
ProgVGAT demonstrates that the key to mastering table-based fact verification lies in bridging the gap between symbolic logic and natural language. By "verbalizing" logical steps and employing graph-based attention, the model can reason more transparently and accurately. While challenges remain regarding template dependency and scalability to massive tables, ProgVGAT represents a significant step toward more robust and interpretable automated fact-checking.
Summary updated