Abstract
The CCKS2019 shared task was devoted to inter-personal relationship extraction. Given two person entities and at least one sentence containing these two entities, participating teams are asked to predict the relationship between the entities according to a given relation list. This year, 358 teams from various universities and organizations participated in this task. In this paper, we present the task definition, the description of data and the evaluation methodology used during this shared task. We also present a brief overview of the various methods adopted by the participating teams. Finally, we present the evaluation results.
Introduction
Relation Extraction (RE) is a fundamental task in information extraction, aimed at transforming unstructured text into structured knowledge graphs. While general RE has seen significant progress, extracting inter-personal relationships presents unique challenges. These relationships are often nuanced, deeply embedded in social contexts, and typically suffer from a lack of high-quality labeled datasets.
Supervised learning models require massive amounts of data, which are expensive to annotate manually. To address this, researchers often employ distant supervision—automatically labeling sentences by aligning them with existing knowledge bases. However, this method introduces "noise," as not every sentence mentioning two entities necessarily describes their known relationship. The CCKS 2019 shared task was designed to evaluate model performance in identifying 34 distinct human relationships amidst such noisy data.
Methodology
The competition was structured into two distinct tracks:
- Sent-Track: Models predict the relationship based on a single sentence.
- Bag-Track: Models predict the relationship based on a "bag" (collection) of all sentences mentioning a specific entity pair.
While a CNN-based baseline was provided, participating teams leveraged state-of-the-art techniques to improve performance:
- Pre-trained Models: The majority of top-performing teams utilized BERT to generate contextualized word representations.
- Feature Engineering: Teams incorporated domain-specific features, such as predicting gender from Chinese names or using surname matching to identify blood relations (e.g., father-son).
- Data Augmentation: Techniques like "back-translation" (translating Chinese to another language and back) were used to expand training data and improve robustness.
- Track Conversion: Some teams trained models on bags and applied the learned patterns to the sentence-level track, as bag-level extraction is inherently more resistant to noise.
Dataset: The IPRE Corpus
The task introduced the Inter-Personal Relationship Extraction (IPRE) dataset, covering 34 relation types plus a "NA" (no relation) category. The training set (70%) was generated via distant supervision using person lists from Wikipedia and Baidu Baike. To ensure rigorous evaluation, the development (10%) and test (20%) sets were manually annotated.
The dataset exhibits a significant "long tail" distribution, where a few "head" relations (e.g., husband, wife, father, mother) account for the majority of the data, while others (e.g., husband's father) have very few samples, making them difficult for models to learn effectively.
Results and Evaluation
A total of 358 teams participated, with top entries significantly outperforming the baseline. Performance was measured using F1 scores (A-score and B-score).
In the Sent-Track, the top F1 score reached 54.3% (Team: 格物致知), a substantial improvement over the baseline's 20.5%. In the Bag-Track, the best F1 score was 63.0% (Team: LEKG), compared to the baseline's 33.2%. The higher performance in the Bag-Track confirms that aggregating multiple context points helps filter out noise from individual sentences.
Discussion and Key Insights
Analysis of the results revealed several critical insights into the current state of inter-personal RE:
- Relationship Difficulty: Models struggled with the "lover" relation, often misclassifying it as "NA" or "friend" despite the presence of clear keywords like "fall in love."
- Classification Bias: There was a notable tendency to over-predict the "father" relation, often funneling general parent-child dynamics into this specific category.
- Domain Knowledge: Heuristics like gender and surname matching remain powerful supplements to deep learning models in the domain of inter-personal relationships.
Limitations and Future Work
The primary challenge remains the extreme class imbalance and the "long tail" of rare relations. Future research should focus on:
- Advanced Denoising: Improving methods to handle false positives from distant supervision.
- Few-Shot Learning: Developing techniques to identify relationships with very few training examples.
- System Fusion: The potential for ensemble models is high; the authors noted that combining the top 10 systems could theoretically achieve an F1 score near 90%, suggesting that different models capture different linguistic nuances.
Summary updated