KIQA: Knowledge-Infused Question Answering Model for Financial Table-Text Data

Rungsiman Nararatwong, Natthawut Kertkeidkachorn, Ryutaro IchiseACL 2022

TL;DR

This paper introduces KIQA, a novel question-answering model designed for financial documents that combine tables and text. By infusing external knowledge using entity linking models like LUKE and GENRE, KIQA significantly improves performance on textual questions, demonstrating the power of symbolic knowledge integration.

What Problem Does This Paper Address?

Question answering (QA) systems have made great strides in handling unstructured text. However, real-world financial documents present a unique challenge: they often contain a hybrid structure of tables and natural language text. Answering questions from such data requires not only understanding the text but also extracting and reasoning over information presented in tables, often involving numerical calculations and financial concepts. Existing entity-linking models, while powerful, struggle to effectively process tabular data. This paper aims to bridge this gap by exploring how to inject external knowledge, specifically entity information, into a QA model to enhance its performance on this complex financial table-text data.

Methods

The core of this work is the KIQA (Knowledge-Infused Question Answering) model, built upon the TagOp baseline. KIQA's novelty lies in its integration of external knowledge through entity linking. The process involves:

  1. Entity Linking: The GENRE model is used to identify and retrieve entities (e.g., company names, financial terms) from the text.
  2. Knowledge Injection: These retrieved entities are then fed into LUKE, a pre-trained language model with entity-aware self-attention. LUKE's architecture is designed to model relationships between words and entities, allowing it to leverage this injected knowledge.
  3. Hybrid QA Architecture: KIQA retains TagOp's three main components: Evidence Extraction (predicting answer spans), Reasoning (performing operations like arithmetic), and Knowledge Injection. The model was adapted to use LUKE for generating knowledge-infused representations.

To understand the impact of this injection, the authors also developed a "KIQA Text" variant that focuses solely on textual input, allowing for a comparison of performance on textual vs. tabular data.

Datasets

The primary dataset used in this research is TAT-QA (Tabular And Textual dataset for Question Answering). This dataset is specifically designed for financial QA tasks and comprises questions, tables, and relevant paragraphs from financial documents. It's characterized by its hybrid data structure and the need for financial knowledge and numerical reasoning. The dataset is split into training (80%), development (10%), and testing (10%) sets.

Key Insights and Ideas

  • External Knowledge Boosts Textual QA: Injecting external entity information significantly improves the model's performance on questions where the answers are found in the textual parts of the documents.
  • Tabular Data Remains a Challenge: While beneficial for text, the current entity linking methods struggle with tabular data, leading to underperformance on table-heavy questions. The external knowledge, when linked to text, can sometimes interfere with the model's ability to process tabular information effectively.
  • Attention Reveals Focus: Attention weight analysis shows that LUKE, when infused with external knowledge, directs its attention more towards relevant textual sections, effectively guiding the model's focus. This is particularly evident when answers are in paragraphs.
  • Symbolic Knowledge is Key: The study reinforces the idea that symbolic knowledge, when properly integrated, can enhance language models' understanding of domain-specific concepts and improve their reasoning capabilities.

Key Results

  • The KIQA model, by injecting external knowledge, achieved an overall improvement of 1.57% compared to the baseline.
  • Performance on textual data saw a substantial improvement of 8.86% for single-span predictions.
  • LUKE & GENRE (L&G) consistently outperformed RoBERTa in arithmetic operations and single-span prediction, highlighting the benefit of entity-aware attention.
  • However, the models underperformed on tabular data, with some configurations showing a decrease in performance compared to the baseline.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Tabular Entity Linking: A significant limitation is the current inability of entity linking models like GENRE to effectively handle tabular data. Future work should focus on developing entity-linking models specifically designed for tabular structures.
  • Irrelevant Entity Interference: The study observed that unrelated entity information can negatively impact the model's performance, especially on tabular data. Finding solutions to mitigate this interference is crucial.
  • Data Variance: The high variance in performance on counting columns is attributed to the small number of samples in this category, suggesting a need for more diverse training data.
  • Hybrid Structure Integration: More efficient methods for integrating tabular and textual symbolic knowledge are needed to fully leverage the hybrid nature of financial documents.

Summary updated

BibTeX
@inproceedings{nararatwong2022kiqa,
  author = {Rungsiman Nararatwong and Natthawut Kertkeidkachorn and Ryutaro Ichise},
  title = {{KIQA: Knowledge-Infused Question Answering Model for Financial Table-Text Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure