ELCO3: Entity Linking with Corpus Coherence Combining Open Source Annotators

Pablo Ruiz, Thierry Poibeau, Frédérique MélanieNAACL 2015

TL;DR

Entity linking (EL) systems struggle with inconsistent performance across different corpora. This paper introduces ELCO3, a novel workflow that combines multiple open-source EL systems using a weighted voting scheme to improve accuracy. The system also provides a user interface that helps researchers assess annotation quality and make informed choices about entity selection for their analyses.

What Problem Does This Paper Address?

Entity Linking (EL) is the task of identifying mentions of entities in text and linking them to a knowledge base (like Wikipedia). However, existing EL systems often perform unevenly, with accuracy varying significantly depending on the type of text (corpus) and the specific entities being linked. This inconsistency makes it difficult for researchers, particularly in social sciences who need to analyze large text collections, to choose the best EL system or trust its output. They need a way to combine the strengths of different systems and to understand the quality of the resulting annotations.

Methods

ELCO3 tackles this problem through a multi-stage workflow:

  1. Combining Open-Source Annotators: The core of the system involves integrating outputs from several publicly available EL tools (Tagme 2, DBpedia Spotlight, and Wikipedia Miner).
  2. Weighted Voting Scheme: To resolve conflicting annotations from different systems, ELCO3 employs a novel weighted voting mechanism. This scheme assigns weights to each annotator based on their historical performance on corpora similar to the user's input. This is inspired by the ROVER method used in speech recognition.
  3. Entity Type Assignment: Entities are categorized into predefined types (Organization, Person, Location, Concept) based on information provided by the EL systems.
  4. Entity Coherence Measures: Beyond simple confidence scores, ELCO3 calculates two measures of an entity's coherence within the corpus:
  5. Wikipedia Link-based Measure (WLM): Assesses the semantic relatedness of an entity to other entities in the corpus, using Wikipedia link structures.
  6. Wikipedia Category Graph Distance: Measures the distance between an entity's Wikipedia categories and those of other representative entities in the corpus.
  7. User Interface (UI): A prototype UI allows users to navigate annotated corpora, view entities and their associated metrics (confidence, coherence), and examine the documents where entities are mentioned. This helps users visually assess annotation quality and make informed decisions.

Datasets

The paper does not explicitly list specific datasets used for training or benchmarking the ELCO3 workflow itself. Instead, it refers to the use of "reference corpora" to evaluate the performance of individual EL systems and to inform the weighting scheme. The demo utilizes a corpus related to the 2008 financial crisis.

Key Insights and Ideas

  • Ensemble Power: Combining multiple EL systems, even off-the-shelf ones, can lead to more robust and accurate entity linking than relying on a single system.
  • Context-Aware Weighting: The performance of an EL system is not static; weighting their contributions based on their past performance on similar data is crucial for effective ensemble methods.
  • Beyond Confidence: Simple confidence scores from EL systems are insufficient. Incorporating measures of entity coherence within the corpus provides a deeper understanding of annotation validity, especially for spotting incorrect or out-of-context links.
  • User-Centric Evaluation: Providing researchers with a UI that visualizes both annotations and their quality metrics empowers them to critically evaluate and select entities for their downstream tasks.

Key Results

The paper demonstrates the utility of ELCO3 through qualitative examples:

  • Spotting Incorrect Annotations: The coherence score effectively helps identify entities that are incorrectly linked, such as "Nielsen ratings" being mistaken for "credit ratings" in a financial crisis corpus. The incorrect entity had a significantly lower coherence score.
  • Verifying Entity Correctness in Networks: In co-occurrence network analysis, ELCO3's coherence scores helped distinguish between a relevant entity (Continental Illinois bank) and a less relevant one (Continental Airlines) for a corpus on the 2008 financial crisis, even when frequency and confidence scores were similar.
  • Complementarity of Linkers: The examples show how different EL systems can miss entities that others identify, highlighting the benefit of combining their outputs.

Limitations and Future Work

  • Current Demo Limitations: The current prototype does not allow users to tag new corpora directly; it operates on pre-processed content.
  • Automatic Visualization: The authors acknowledge that the workflow does not automatically generate visualizations like co-occurrence networks. This is identified as a valuable future direction for their target users.
  • Sentence-Level Networks: The workflow's ability to segment documents into sentences opens up possibilities for creating co-occurrence networks at a finer granularity, which is a potential area for future exploration.

Summary updated

BibTeX
@inproceedings{ruiz2015elco,
  author = {Pablo Ruiz and Thierry Poibeau and Frédérique Mélanie},
  title = {{ELCO3: Entity Linking with Corpus Coherence Combining Open Source Annotators}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure