AffilGood: Building reliable institution name disambiguation tools to improve scientific literature analysis

Nicolau Duran-Silva, Pablo Accuosto, Piotr Przybyła, Horacio SaggionACL 2024

TL;DR

This paper introduces AffilGood, a framework and suite of tools designed to tackle the persistent challenge of accurately identifying and disambiguating research institutions from messy, unstructured affiliation strings in scientific publications. By releasing new annotated datasets and fine-tuned language models, AffilGood aims to significantly improve the reliability of scientific literature analysis and research assessment.

What Problem Does This Paper Address?

Accurately attributing scientific works to the correct research institutions is fundamental for understanding research trends, evaluating performance, and building comprehensive knowledge graphs. However, this is a surprisingly difficult task due to the "affiliation normalization" problem. Authors often provide affiliation strings in non-standard formats, using abbreviations, different languages, and including extraneous information like department names or even email addresses. Existing methods struggle with this noise, lack of standardization, and the sheer volume of variations, especially in multilingual contexts. Furthermore, there's a scarcity of high-quality, manually annotated data to train and evaluate these disambiguation systems, hindering progress in the field.

Methods

AffilGood tackles institution name disambiguation through a three-step pipeline:

  1. Affiliation Span Identification: This initial step aims to clean up raw affiliation strings by identifying and extracting only the relevant parts that actually describe an institution, filtering out noise like email addresses or acknowledgments.
  2. Named Entity Recognition (NER): A crucial step where models identify and classify specific entities within the cleaned affiliation strings, such as organization names, cities, countries, and addresses. The authors fine-tuned RoBERTa and XLM-RoBERTa models, specifically adapting them with further pre-training on raw affiliation strings to better understand their unique grammatical structure.
  3. Entity Linking: This final step connects the identified entities (organizations and locations) to unique, persistent identifiers from global registries, primarily the Research Organization Registry (ROR). The paper explores three linking strategies: the S2AFF linker, a basic Elasticsearch search, and a more sophisticated approach combining Elasticsearch with a quantized generative model for re-ranking.

Datasets

The paper introduces several new datasets to address the data scarcity problem:

  • Affiliation Span Identification Dataset: Annotated with affiliation spans from noisy, automatically extracted strings.
  • NER Dataset: Annotated with named entities (organizations and locations) from a large set of affiliation strings.
  • Seven Entity Linking Datasets: These datasets are designed to evaluate entity-linking systems, varying in difficulty and including:
  • Raw affiliation strings: Multilingual Affiliations (MA), French Affiliations (FA), Non-Related Multi-Organizations (NRMO), and Mixed Affiliations (S2AFF*).
  • Pre-segmented entities: CORDIS, ETER English (ETERe), and ETER Multilingual (ETERm), which come from R&I projects and statistical databases.

These datasets are made publicly available to foster further research.

Key Insights and Ideas

  • The "noise" in affiliation strings is a major hurdle: Simply extracting text from publications is insufficient; robust filtering and identification of relevant spans are critical first steps.
  • Adapting language models to the "grammar" of affiliations matters: Fine-tuning models like RoBERTa and XLM-RoBERTa on raw affiliation strings themselves (further pre-training) significantly improves their ability to understand and process these specific types of text, outperforming base models.
  • A multi-stage approach is essential: Combining NER with sophisticated entity linking, especially with a re-ranking mechanism, is key to achieving high accuracy, particularly for complex and multilingual affiliations.
  • Publicly available, high-quality datasets are vital: The creation and release of these new annotated datasets are a significant contribution, enabling more rigorous evaluation and development in the field.

Key Results

  • The adapted NER models (AffilRoBERTa and AffilXLM) achieved strong performance, with AffilXLM-NER showing particular strength on multilingual datasets.
  • The proposed AffilGood pipeline, combining the NER models with an Elasticsearch-based entity linker and a generative re-ranking step, consistently outperformed baseline methods across various datasets.
  • For complex French affiliations, the AffilGood pipeline achieved an 87% gain in macro-averaged F1-score compared to the full S2AFF pipeline when replacing its NER component with AffilGood's models.
  • The entity linking component, particularly the Elasticsearch + qLLM approach, achieved high F1 scores (e.g., 0.936 on ETERm), demonstrating its effectiveness.

Limitations and Future Work

  • Span identification evaluation: The paper notes that the span identification task was not fully integrated into the final pipeline evaluation due to a limited number of examples where affiliation spans did not cover the entire raw string. This suggests future work could focus on creating more challenging span identification datasets.
  • Multilingual complexity: While the models show promise in multilingual settings, the paper acknowledges that affiliations can be in diverse languages and use mixed language, indicating ongoing challenges.
  • Future work could explore: Further improvements to the NER and entity linking modules, potentially incorporating more sophisticated cross-lingual techniques or exploring other global registries beyond ROR. The authors also emphasize the value of their released resources for future research.

Summary updated

BibTeX
@inproceedings{duransilva2024affilgood,
  author = {Nicolau Duran-Silva and Pablo Accuosto and Piotr Przybyła and Horacio Saggion},
  title = {{AffilGood: Building reliable institution name disambiguation tools to improve scientific literature analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure