TL;DR
This paper introduces "Statements," a novel, domain-agnostic data structure for universal information extraction from tables, particularly for ESG (Environmental, Social, and Governance) KPIs. They propose "Statement Extraction" as a new supervised deep-learning task and present SemTabNet, a large dataset of annotated ESG tables, achieving state-of-the-art results with their T5-based models.
What Problem Does This Paper Address?
Extracting quantitative information from tables in ESG reports is a significant challenge due to the high variability in table structures and content. While Large Language Models (LLMs) excel at text-based information extraction, their performance degrades when applied directly to tables. This makes it difficult for organizations and researchers to systematically analyze the vast amounts of ESG data, hindering efforts to understand corporate performance on critical environmental, social, and governance issues.
Methods
The core innovation is the "Statement" data structure, a tree-like representation that normalizes diverse tabular information into a uniform, domain-agnostic format. This structure consists of "statement nodes" and "predicate nodes" that capture subjects, properties, values, and units.
The authors then frame information extraction from tables as a "Statement Extraction" task. They fine-tune T5 encoder-decoder transformer models for this task, exploring three main experimental setups:
- SE Direct: The model receives the table in markdown format and directly outputs statements in markdown.
- SE Indirect 1D: The model processes individual table cells sequentially to predict labels, which are then used to construct statements.
- SE Indirect 2D: The model processes the entire table in markdown to predict cell labels simultaneously, which are then used to construct statements.
To evaluate the quality of extracted statements, they propose a "Tree Similarity Score" based on Tree Edit Distance, measuring the structural similarity between predicted and ground-truth statement trees.
Datasets
The paper introduces SemTabNet, a novel dataset comprising over 100,000 annotated tables extracted from ESG reports. Each cell in these tables is annotated to reflect its semantic relationship with other cells, enabling the training and evaluation of statement extraction models.
Key Insights and Ideas
- Statements as a Universal Normalizer: The "Statement" structure proves effective in unifying information from heterogeneous tables across different domains, making it a powerful tool for cross-domain analysis.
- Indirect Statement Extraction Outperforms Direct: Surprisingly, the indirect methods (SE Indirect 1D and 2D), where models first predict cell labels, significantly outperform direct statement extraction, especially SE Indirect 2D. This suggests that breaking down the complex task into intermediate steps (label prediction) is more effective for LLMs.
- Tree Similarity Score as a Robust Metric: The proposed Tree Similarity Score provides a nuanced evaluation of statement extraction, capturing both entity and relationship extraction quality, which is crucial for complex data structures.
- LLMs Struggle with Direct Table-to-Statement Translation: The poor performance of LLMs in direct statement extraction highlights the inherent challenges of translating dense, structured tabular data into a sequence of statements without intermediate representations.
Key Results
The authors' best-performing fine-tuned T5 model achieved a Tree Similarity Score of 82%, a significant improvement over baseline LLMs (average 0-21%) using in-context learning. For entity extraction, the models achieved average F1 scores of over 95% in the SE Indirect 2D task. The SE Indirect 2D approach, in particular, demonstrated strong performance, with the SEM-T5-3b model achieving an 81.76% Tree Similarity Score.
Limitations and Future Work
- Domain Specificity: While the "Statement" structure is domain-agnostic, the current work focuses on ESG reports. The authors suggest extending this to other domains.
- Declarative and Quantitative Knowledge: The current framework is limited to declarative, explicit, and quantitative knowledge, excluding qualitative or implicit information.
- Model Training Bias: The training strategy was biased against larger models, potentially impacting the full potential of larger LLMs.
- Future Work: The authors propose extending the framework to multi-modality (including text data) and exploring its use in downstream tasks like question-answering and document summarization. They also aim to improve zero-shot statement extraction capabilities.
Summary updated