Overview of the CCKS 2019 Knowledge Graph Evaluation Track: Entity, Relation, Event and QA

Xianpei Han, Zhichun Wang, Jiangtao Zhang, Qinghua Wen, Wenqi Li, Buzhou Tang, Qi Wang, Zhifan Feng, Yang Zhang, Yajuan Lu, Haitao Wang, Wenliang Chen, Hao Shao, Yubo Chen, Kang Liu, Jun Zhao, Taifeng Wang, Kezun Zhang, Meng Wang, Yinlin Jiang, Guilin Qi, Lei Zou, Sen Hu, Minhao Zhang, Yinnian LinarXiv 2020 · OpenReview

TL;DR

The CCKS 2019 Knowledge Graph Evaluation Track brought together over 1600 teams to tackle six diverse tasks in knowledge graph construction and application, spanning entity, relation, event extraction, and question answering. This paper provides a comprehensive overview of these tasks, highlighting the methodologies, datasets, and key insights that shaped the state-of-the-art in Chinese knowledge graph research for 2019.

What Problem Does This Paper Address?

Knowledge graphs (KGs) are crucial for modeling world knowledge, enabling applications from information retrieval to question answering. However, advancing KG technology requires standardized evaluation and understanding of current capabilities, challenges, and future directions. CCKS 2019 aimed to address this by organizing a large-scale evaluation track with six distinct tasks, covering fundamental KG technologies and their applications, particularly within the Chinese language context. The paper serves to document and analyze the outcomes of this evaluation, providing a valuable reference for researchers and practitioners.

Methods

The paper details the methodologies employed across six tasks:

  • Medical Named Entity Recognition (NER) and Attribute Extraction: Primarily addressed using sequence labeling models like CRF and BiLSTM, with a significant trend towards incorporating pre-trained language models (BERT, ELMo) and feature engineering. A key challenge was the cross-domain transfer learning for attribute extraction.
  • Short Text Entity Linking: This task involved entity recognition and disambiguation. Common approaches included pipeline methods (NER + disambiguation) and end-to-end models. Pre-trained models like ERNIE/BERT were widely adopted, and knowledge enhancement through external resources was a key strategy.
  • Person Relation Extraction: This task was split into sentence-level (Sent-Track) and bag-level (Bag-Track) extraction. Methods ranged from supervised learning to distant supervision. The use of BERT and model ensemble techniques were prevalent. Feature engineering, particularly leveraging linguistic cues like gender and surname, played a significant role.
  • Financial Event Subject Extraction: This task was framed as either a sequence labeling or reading comprehension problem. BERT-based models were dominant, with a strong emphasis on model ensemble for robustness. The diversity of language expression and the challenge of multiple events within a single text were key difficulties.
  • Public Company Announcement Information Extraction: This involved extracting information from both structured tables and unstructured text within PDF documents. Approaches included PDF-to-image conversion, table detection/extraction (e.g., using OpenCV, Faster R-CNN), and sequence labeling or reading comprehension models for text extraction.
  • Chinese Knowledge Graph Question Answering (KBQA): This task focused on answering factual questions using a given knowledge base (PKUBASE). Methods involved entity recognition, relation extraction, and path matching. Models often employed a modular approach, with a trend towards using BERT for improved entity and relation recognition.

Datasets

The paper describes datasets used for each task:

  • Task 1 (Medical NER): Datasets derived from Chinese electronic medical records, with annotations for diseases, treatments, symptoms, and drugs.
  • Task 2 (Entity Linking): A knowledge base derived from Baidu Baike (approx. 390k entities) and annotated short text data from internet queries.
  • Task 3 (Person Relation Extraction): The IPRE (Inter-Personal Relationship Extraction) dataset, a Chinese dataset specifically for person relation extraction, with both automatically generated and manually annotated data.
  • Task 4 (Event Subject Extraction): Financial news corpus with annotated event types and subjects.
  • Task 5 (Announcement Extraction): Public company annual reports (PDFs) and "personnel change" announcement documents, with structured information extracted.
  • Task 6 (KBQA): A dataset built upon the PKUBASE knowledge graph, featuring Chinese natural language questions and their corresponding SPARQL queries and answers.

Key Insights and Ideas

  • Pre-trained Language Models (PLMs) are Dominant: Across almost all tasks, BERT, ELMo, and similar PLMs significantly boosted performance, becoming a de facto standard for state-of-the-art results.
  • Model Ensemble is Crucial for Robustness: Many top-performing teams utilized model ensembles to improve accuracy and robustness, especially in tasks with complex data or noisy labels.
  • Domain Knowledge is Key for Specific Tasks: For specialized domains like medical or financial text, incorporating domain-specific dictionaries, features, or pre-training on domain corpora (e.g., caBERT for financial news) yielded substantial improvements.
  • Challenges in Cross-Domain and Low-Resource Settings: Tasks involving domain transfer (e.g., medical attribute extraction) or limited labeled data (e.g., entity linking with noisy distant supervision) remained significant challenges.
  • PDF Information Extraction is Non-Trivial: Extracting structured information from PDFs proved difficult due to their lack of inherent structure, requiring sophisticated techniques for layout analysis and content extraction.
  • Chinese Language Specifics Pose Challenges: Tasks involving Chinese text highlighted unique challenges like the absence of explicit word segmentation and the ambiguity of natural language.

Key Results

  • High Performance Achieved: Top-performing systems demonstrated impressive F1 scores across various tasks, often exceeding 0.85 for medical NER, 0.76 for entity linking, and reaching over 0.54 for person relation extraction. Financial event extraction saw F1 scores above 0.93, and announcement extraction achieved scores above 0.95. KBQA results showed top teams achieving F1 scores around 0.73.
  • PLMs Outperform Traditional Models: The widespread adoption of PLMs led to significant performance gains compared to previous generation models like LSTMs and CRFs.
  • Ensembling Boosts Scores: Model ensembles consistently improved performance, with top teams often integrating multiple models.
  • Domain-Specific Models Show Promise: For instance, a domain-specific pre-trained model (caBERT) for financial news significantly improved performance in entity linking.

Limitations and Future Work

  • Low-Resource Scenarios: The paper highlights the need for further research into low-resource KG technologies, especially for building high-performance systems with limited labeled data, given the cost of annotation.
  • Domain and Data Shift Robustness: A significant performance drop was observed when training and testing data came from different scenarios. Future work should focus on developing robust KG techniques that can handle domain, corpus, and task variations, such as through transfer learning and unsupervised learning.
  • Chinese Language Nuances: The unique characteristics of the Chinese language (e.g., lack of word segmentation) continue to pose challenges for KG technologies, requiring specialized approaches.
  • PDF Extraction Efficiency: While methods for PDF information extraction were presented, improving the speed and efficiency of parsing and extraction remains a challenge.
  • End-to-End KBQA: While modular systems were prevalent in KBQA, the paper suggests that end-to-end models might offer benefits in discovering hidden patterns, especially as datasets grow.
  • Handling Complex Questions in KBQA: The tendency to limit query paths to 2 hops in KBQA might limit performance in real-world open-domain scenarios, suggesting a need for models that can handle more complex questions.

Summary updated

Abstract

Knowledge graph models world knowledge as concepts, entities, and the relationships between them, which has been widely used in many real-world tasks. CCKS 2019 held an evaluation track with 6 tasks and attracted more than 1,600 teams. In this paper, we give an overview of the knowledge graph evaluation tract at CCKS 2019. By reviewing the task definition, successful methods, useful resources, good strategies and research challenges associated with each task in CCKS 2019, this paper can provide a helpful reference for developing knowledge graph applications and conducting future knowledge graph researches.
BibTeX
@misc{han2020overview,
  author = {Xianpei Han and Zhichun Wang and Jiangtao Zhang and Qinghua Wen and Wenqi Li and Buzhou Tang and Qi Wang and Zhifan Feng and Yang Zhang and Yajuan Lu and Haitao Wang and Wenliang Chen and Hao Shao and Yubo Chen and Kang Liu and Jun Zhao and Taifeng Wang and Kezun Zhang and Meng Wang and Yinlin Jiang and Guilin Qi and Lei Zou and Sen Hu and Minhao Zhang and Yinnian Lin},
  title = {{Overview of the CCKS 2019 Knowledge Graph Evaluation Track: Entity, Relation, Event and QA}},
  year = {2020},
  eprint = {2003.03875},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2003.03875},
}
Figure