TL;DR
Information extraction (IE) systems often produce factual errors. This paper introduces a novel framework that jointly identifies and corrects inconsistencies between extracted beliefs and their source text, improving both accuracy and robustness. By treating consistency and repair as interconnected tasks, the approach achieves significant performance gains over existing methods.
What Problem Does This Paper Address?
Information extraction (IE) systems aim to automatically extract structured knowledge (entities and relationships) from unstructured text, often representing this knowledge in a knowledge graph. However, current IE systems are far from perfect, with significant error rates. One major source of error is inconsistency: the extracted "belief" (e.g., a subject-relation-object triple) doesn't accurately reflect the information present in the source text (provenance). This inconsistency can arise from various issues, including misinterpretations by the IE system, noisy extraction processes, or even errors in the source text itself. While previous work has focused on detecting inconsistencies or improving IE system accuracy in general, this paper tackles the problem of jointly identifying these inconsistencies and fixing them, offering a more comprehensive solution.
Methods
The proposed framework consists of three main modules:
- Representation: Both the extracted belief (subject, relation, object) and its associated provenance sentences are embedded into vector spaces. The belief is represented by a single vector, while provenance sentences are processed at both individual sentence-level and a combined "blob"-level. Various embedding techniques are explored, including bag-of-words (BoW), LSTMs, and BERT.
- Combination: The representations of the belief and provenance are combined. A key technique here is a similarity-based attention mechanism that scores provenance sentences based on their relevance to the belief. These scores are used to create a weighted average of the provenance representations, forming a combined representation that captures the semantics of both the belief and its supporting text.
- Feature Learning and Classification: The combined representation is then fed into a multi-layer perceptron (MLP) for feature learning. This learned representation is used to train two classifiers: one for binary consistency prediction (consistent/inconsistent) and another for categorical repair (predicting the correct relation if inconsistent). The framework is trained using a joint optimization objective, combining the losses from both tasks. This joint training allows the models to learn shared representations that benefit both consistency detection and repair.
Datasets
The authors evaluate their approach on three datasets:
- TAC 2015 and TAC 2017: These datasets are derived from the Text Analysis Conference (TAC) Knowledge Base Population shared tasks. They contain beliefs extracted by numerous IE systems, each supported by up to four provenance sentences. Consistency labels were provided by human experts.
- TACRED-KG: This is a novel dataset created by adapting the existing TACRED relation extraction dataset. It provides provenance-backed beliefs and uses the gold standard relation as the "repair label."
Key Insights and Ideas
- Joint Consistency and Repair is Crucial: Treating consistency detection and belief repair as separate tasks is suboptimal. By jointly learning them, the framework leverages shared representations, leading to better performance on both.
- Lexical Features Can Be Sufficient: Surprisingly, simple lexical features (like TF-IDF) combined with sentence-level attention and a basic MLP often outperform more complex sequential models (like LSTMs or BERT) for both consistency and repair, especially on noisier datasets. This suggests that for these tasks, the explicit mention of entities and relations in the text is highly informative.
- Credibility Models Don't Directly Transfer: Existing models designed for general fact-checking or credibility assessment perform poorly when directly applied to the consistency and repair tasks, highlighting the unique challenges of this problem.
- Provenance Quality Matters: The TACRED-KG dataset, with its more curated provenance, allows for better performance compared to the TAC datasets, where provenance can be noisier and less complete. This emphasizes the importance of provenance quality in IE.
Key Results
The proposed framework consistently outperforms strong baselines, including existing credibility models, across all three datasets.
- Consistency: The best models achieved F1 scores in the high 60s to low 70s on TACRED-KG and TAC 2017, and in the high 40s to low 50s on TAC 2015.
- Repair: The framework achieved macro-averaged F1 scores in the high 70s to low 80s on TAC 2015 and TAC 2017, and in the high 60s to low 70s on TACRED-KG.
- The use of TF-IDF sparse features combined with sentence attention and an MLP was particularly effective, often surpassing more complex embedding methods.
Limitations and Future Work
- Dataset Differences: The performance varies significantly across datasets, with TACRED-KG showing better results due to its more controlled provenance. This suggests that the framework's effectiveness is sensitive to the quality and characteristics of the provenance data.
- Sequential Modeling: While simple lexical methods performed well, the paper notes that sequence-based encodings can improve precision and F1 for consistency in some cases, suggesting potential for further exploration.
- Repair vs. Re-Extraction: The paper acknowledges that the repair task can be viewed as relation re-extraction, and the performance drop on TACRED-KG compared to TAC datasets highlights differences in dataset construction assumptions. Future work could explore more sophisticated re-extraction strategies.
- The authors suggest that their framework is generalizable and can be adapted to different IE system outputs and schemas, indicating potential for broader application.
Summary updated