A Study on Entity Linking Across Domains: Which Data is Best for Fine-Tuning?

Hassan Soliman, Heike Adel, Mohamed H. Gad-Elrab, Dragan Milchevski, Jannik StrötgenACL 2022

TL;DR

This paper investigates how to effectively fine-tune neural entity linking systems for new domains by creating a joint vector space for entities from multiple knowledge graphs (KGs). They found that combining data from both the general (source) and specific (target) domains is crucial for maintaining performance across both, and that overlapping entities, while good for intrinsic evaluation, have less impact on the final linking task.

What Problem Does This Paper Address?

Entity linking (EL) is a fundamental task in natural language processing, aiming to identify and disambiguate mentions of entities in text by linking them to entries in a knowledge graph (KG). While EL systems have advanced significantly, extending them to specialized domains where annotated data is scarce remains a challenge. This paper tackles the problem of building a unified entity linking system that can link mentions to entities from both a general-domain KG (like Wikipedia) and a domain-specific KG simultaneously. A key question is how to best leverage available data for fine-tuning such a system, particularly when dealing with limited data in the target domain and the potential for performance degradation on the general domain.

Methods

The authors build upon the BLINK (Wu et al., 2020) neural entity linking model, which uses a three-part architecture: a context encoder, a candidate encoder, and a cross-encoder. The process involves candidate generation (finding potential entities for a mention) and candidate ranking (scoring the likelihood of a mention-entity pair).

To extend this to new domains, they focus on fine-tuning the context and candidate encoders. The core novelty lies in their investigation of different data sources for this fine-tuning process:

  • Target domain data (T): Data annotated with entities from the specific target KG.
  • Source domain data (S): Data annotated with entities from the general source KG (Wikipedia).
  • Overlapping entities (O): Entities that exist in both the source and target KGs.

They propose specific loss functions for fine-tuning on these data sources, particularly for overlapping entities, aiming to align their representations in a joint vector space. The paper also introduces a method for identifying overlapping entities based on string matching and semantic similarity of their descriptions.

Datasets

  • Source Domain KG: Wikipedia (using Reddit data for mentions).
  • Target Domain KGs: Four domains from the Wikia dataset: American Football, Doctor Who, Fallout, and Final Fantasy.
  • Data for Fine-tuning:
    • T: Domain-specific data (split from Wikia datasets).
  • S: Reddit dataset (annotated with Wikipedia entities).
  • O: A newly created list of overlapping entities between Wikipedia and each target domain KG.

Key Insights and Ideas

  • Joint Vector Space is Key: Representing entities from different KGs in a shared vector space is an effective way to handle multi-KG entity linking, especially for overlapping entities.
  • Source + Target Data is Optimal: Fine-tuning with a combination of data from both the source (general) and target (specific) domains yields the best entity linking performance.
  • Overlapping Entities: Good for Intrinsic, Less for Extrinsic: While overlapping entities significantly improve the intrinsic evaluation of the joint vector space (making entity representations closer), their impact on the extrinsic entity linking task is less pronounced. This is likely due to the cross-encoder's ability to re-evaluate pairs using context, mitigating reliance solely on the joint vector space.
  • Source Data Prevents Degradation: Crucially, including data from the source domain (S) prevents performance degradation on the general domain, a significant concern when fine-tuning on specialized data.
  • Domain-Specific Data is Essential: Fine-tuning solely on source data (S) improves performance on the source KG, but fine-tuning solely on target data (T) can harm performance on the source KG.

Key Results

  • Intrinsic Evaluation: Fine-tuning, especially on overlapping entities (O), significantly improves the quality of the joint vector space, as measured by Average Cosine Similarity (ACS) and Mean Reciprocal Rank (MRR), outperforming the baseline BLINK model in most cases.
  • Extrinsic Evaluation (Target Domain): The best performance on the target domain KGs is achieved by fine-tuning on a combination of source and target data (TS/TOS), leading to improvements of up to 20% in MAP@10 and AP@1 compared to the baseline.
  • Extrinsic Evaluation (Source Domain): Fine-tuning on source data (S) improves performance on the source KG test set. Combining source and target data (TS/TOS) also enhances performance on the source KG, effectively avoiding performance loss.

Limitations and Future Work

  • Overlapping Entity Quality: While they filtered overlapping entities based on semantic similarity, manual checks revealed some incorrect or unclear pairs, suggesting room for improvement in the overlap extraction process.
  • No New Benchmark: The paper notes the lack of a readily available benchmark for domain-specific entity linking data and uses a split of the Wikia dataset, highlighting a need for standardized datasets.
  • Future Work: The authors suggest that their findings provide a robust system for linking mentions to both source and target-domain KGs simultaneously. Further research could explore more sophisticated methods for identifying and leveraging overlapping entities or investigate other data augmentation strategies.

Summary updated

BibTeX
@inproceedings{soliman2022study,
  author = {Hassan Soliman and Heike Adel and Mohamed H. Gad-Elrab and Dragan Milchevski and Jannik Strötgen},
  title = {{A Study on Entity Linking Across Domains: Which Data is Best for Fine-Tuning?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure