A practical and nuanced framework for entity linking evaluation

Fuqi Xu, Goran Nenadić, Robert StevensJournal of Biomedical Semantics 2026 · DOI

TL;DR

This paper introduces a novel framework for evaluating entity linking (EL) systems, moving beyond simple accuracy to capture nuanced model behavior, especially within hierarchical vocabularies. By analyzing match types, distances, and locations within these structures, the framework provides a more practical and application-aligned assessment of EL models.

What Problem Does This Paper Address?

Entity linking (EL) is the task of mapping mentions in text to specific entities in a knowledge base or vocabulary. While accuracy is the go-to metric for evaluating EL systems, it often fails to capture the full picture of a model's performance. This is particularly true in domains like biomedicine, where vocabularies are highly structured with hierarchical relationships (e.g., "disease" -> "infectious disease" -> "viral infection"). Simple accuracy metrics don't account for whether a model correctly identifies the level of specificity required by a particular application. For instance, linking "apple" to "fruit" might be perfectly acceptable in one context, but insufficient in another where a specific cultivar like "Granny Smith" is needed. Furthermore, standard evaluations can be skewed by dataset biases like mention repetition and lack of novel entities, leading to an overestimation of a model's true generalization capabilities.

Methods

The authors propose a comprehensive evaluation framework that addresses these limitations. Key components include:

  • Hierarchical Vocabulary Emphasis: The framework explicitly considers the hierarchical structure of vocabularies. Instead of just exact matches, it categorizes matches into types like "Exact," "Overspecific" (predicted entity is a subset of the target), "Underspecific" (predicted entity is a superset of the target), and "Orthogonal" (no clear hierarchical relationship).
  • Match Distance and Location: Metrics are introduced to quantify the "distance" between the predicted and target entities within the hierarchy and the "location" (relative depth) of the target entity within its branch. This helps understand the severity of mismatches.
  • Performance Profiles: For each mention-entity pair, a detailed profile is generated, capturing its match type, distance, and location. These profiles can be visualized (e.g., as heatmaps) to reveal patterns in model errors.
  • Model Characteristics: The framework summarizes these profiles into higher-level characteristics like "Coverage" (proportion of entities accurately annotated, allowing broader matches) and "Specificity" (granularity of predicted entities). Additional metrics like "Braveness" (tendency for specific predictions) and "Cautiousness" (tendency for general predictions) are also defined.
  • Target Assessment Sets: To address dataset biases, the authors propose using different evaluation sets:
  • Global Set: The entire test set.
  • Global Unique Set: Removes mention-entity pair repetitions.
  • Novel Set: Includes only mention-entity pairs not seen in training/development.
  • Novel Unique Set: Removes repetitions from the Novel Set, offering the most rigorous test of generalization.

Datasets

The proposed framework was evaluated using the BC5CDR disease dataset. The authors also reference other common entity linking datasets like MeSH, OBO, NCBI Taxonomy, Cellosaurus, UMLS, and NLM-Gene in their analysis of dataset characteristics.

Key Insights and Ideas

  • Accuracy is Not Enough: The core idea is that a single accuracy score is insufficient. Models with similar accuracy can exhibit vastly different error patterns, making direct comparison misleading.
  • Hierarchy Matters: In structured domains like biomedicine, understanding how well a model navigates hierarchical relationships is crucial. An underspecific match might be acceptable, while an orthogonal one is likely an error.
  • Generalization is Key: Evaluating on "novel" and "unique" sets is critical for understanding how well models perform on unseen data, which is more representative of real-world applications.
  • Application-Specific Evaluation: The framework emphasizes that the "best" evaluation strategy depends on the specific use case. For example, prioritizing coverage might be important for insurance claims, while precise location might be critical for detailed biological research.

Key Results

  • Both TaggerOne and BioSyn models showed a significant drop in performance (accuracy) on the "novel unique set" compared to the "global set," highlighting the impact of novel entities and repetitions.
  • Over 50% of mismatches for both models were classified as "orthogonal," indicating a common failure to identify the correct entity branch.
  • TaggerOne generally exhibited broader coverage, while BioSyn showed greater predictive "braveness" (more specific predictions). TaggerOne was more "cautious," especially on novel data.
  • The analysis revealed that the proportion of novel entities in a test set significantly influences model accuracy, more so than vocabulary size or training dataset size.

Limitations and Future Work

  • Dataset Size and Confounding Factors: The authors acknowledge that given confounding factors and limited sample sizes in some analyses, conclusions require further investigation.
  • Composite Scores: While the framework clarifies underlying dimensions, future work could explore creating composite scores that aggregate multiple aspects of performance for specific use cases.
  • Broader Vocabulary Integration: The framework's applicability to more complex, integrated vocabularies could be further explored.
  • Task-Specific Metrics: The paper advocates for aligning metrics with specific application requirements, suggesting that future research should focus on developing more tailored evaluation approaches.

Summary updated

Abstract

Entity linking maps textual mentions with entities in vocabularies. While accuracy is the primary metric for entity linking evaluation, it fails to capture the complexity of model behaviour. We propose an entity linking evaluation framework that clarifies the target objects of metrics calculation, incorporates term hierarchy, generates performance profiles, and summarises them as model characteristics. The framework emphasises hierarchical vocabulary structures, shifting the focus from text-level label matching to semantic comparison between concepts. We illustrate the competence and utility of this framework through a case study on disease entity linking. Our results highlight the importance of aligning evaluation metrics with application-specific requirements, and provide structured hierarchical error analysis for entity linking, paving the way for more nuanced and practical assessments of entity linking systems.
BibTeX
@article{xu2026practical,
  author = {Fuqi Xu and Goran Nenadić and Robert Stevens},
  title = {{A practical and nuanced framework for entity linking evaluation}},
  journal = {Journal of Biomedical Semantics},
  year = {2026},
  doi = {10.1186/s13326-025-00339-0},
}
Figure