Results of the second evaluation of matching tools

Christian Meilicke, José Luis López Aguirre, Jérôme Euzenat, Ondřej Šváb-Zamazal, Ian Horrocks, Cássia Trojahn dos SantosHAL 2012

TL;DR

This paper presents the results of the third integrated OAEI/SEALS campaign for evaluating ontology matching tools. It highlights the performance of various tools across diverse datasets, focusing on scalability, precision, recall, and runtime, and reveals that specialized multilingual techniques significantly outperform general approaches.

What Problem Does This Paper Address?

The paper addresses the challenge of automatically finding correspondences between ontologies, a crucial task for data integration and semantic web applications. While many ontology matching tools exist, their performance can vary significantly depending on the dataset and the specific matching task. This evaluation campaign aims to provide a comprehensive and reproducible assessment of these tools, identifying their strengths and weaknesses, and guiding users in selecting the most appropriate tool for their needs. The campaign also explores the scalability of these tools and their ability to handle increasingly complex and multilingual datasets.

Methods

The evaluation campaign employed the SEALS (Semantic Evaluation at Large Scale) infrastructure, which provides a standardized environment for executing and evaluating matching tools. This includes a virtualization infrastructure, a tools repository, and a test data repository. The methodology involved:

  • Standardized Execution: All participating tools were executed within the SEALS virtualization environment, ensuring reproducible results.
  • Diverse Datasets: A range of datasets were used, including established benchmarks (Anatomy, Benchmark, Conference) and new, more challenging datasets (MultiFarm for multilingual matching, Large BioMed for large-scale matching).
  • Comprehensive Evaluation Metrics: Beyond traditional precision and recall, the evaluation also considered runtime, scalability (impact of available cores), and a novel "recall+" metric for the Anatomy track to assess the discovery of non-trivial correspondences.
  • Focus on Scalability: The campaign specifically investigated how well tools scale with increasing ontology size and with the number of available CPU cores.

Datasets

The evaluation utilized the following datasets:

  • Benchmark: Focused on scalability with ontologies of increasing size.
  • Anatomy: Used for evaluating performance and the discovery of non-trivial correspondences.
  • Conference: A dataset with a high variance in structure and size.
  • MultiFarm: A new dataset designed for multilingual ontology matching, featuring test cases with ontologies translated into various languages.
  • Large BioMed: A new dataset comprising very large ontologies from the biomedical domain, used to test the scalability of matching tools.

Key Insights and Ideas

  • Multilingual Matching is Hard, but Specific Techniques Excel: The MultiFarm dataset revealed that systems employing specialized multilingual techniques (like YAM++, AUTOMSv2, and WeSeE) significantly outperform general-purpose matchers when dealing with different languages.
  • No Single "Best" Tool: The evaluation across various tracks showed that different tools excel in different scenarios. Some are better at precision, others at recall, and some are more suited for specific domains (e.g., biomedical).
  • Scalability is Crucial and Varies Widely: While some tools demonstrate good scalability with increasing ontology size and available cores, others struggle, highlighting the need for efficient algorithms.
  • Runtime Doesn't Necessarily Correlate with Quality: The fastest tools don't always produce the best alignments, and vice-versa. This suggests a complex trade-off between speed and accuracy.
  • Reproducibility is Key: The SEALS infrastructure significantly improved the reproducibility of evaluation results, a long-standing challenge in ontology matching research.

Key Results

  • Multilingual Performance: YAM++, AUTOMSv2, and WeSeE achieved the best results on the MultiFarm dataset, with YAM++ often outperforming WeSeE even when using the same translator.
  • Benchmark Performance: MapSSS and Lily provided high-quality alignments on the Benchmark track, though they were among the slowest. LogMapLt, LogMap, Aroma, and GOMMA were among the fastest.
  • Anatomy Results: Agreementmaker and GOMMA-bk led in F-measure for the Anatomy track, with Agreementmaker benefiting from UMLS background knowledge.
  • Conference Results: YAM++, CODI, and LogMap were identified as the best-performing systems on the Conference track.
  • Large BioMed Results: GOMMA-bk achieved the best F-measure on the small module of the Large BioMed dataset, while GOMMA-nobk provided the most precise alignments.

Limitations and Future Work

  • Ambiguous Scalability Results: The scalability experiments for the Anatomy and Large BioMed tracks yielded ambiguous results, with high variance in runtimes and a lack of clear correlation between core count and performance order for some systems.
  • Missing RAM Impact: Systematic experiments on the impact of available RAM on scalability were missing and identified as an important aspect for future work in OAEI 2012.
  • Need for More Diverse Datasets: While new datasets like MultiFarm and Large BioMed were introduced, the authors suggest the continued need for more well-suited datasets.
  • Understanding System Behavior: The paper notes that understanding why certain systems perform unexpectedly (positively or negatively) is difficult without detailed descriptions of their matching strategies or extensions.
  • Coherence of Alignments: The paper highlights the importance of assessing the coherence of generated alignments, noting that even precise mappings can lead to a high amount of unsatisfiable classes. Only a few systems in this campaign utilized techniques for this.

Summary updated

Abstract

meilicke2012b
BibTeX
@inproceedings{meilicke2012results,
  author = {Christian Meilicke and José Luis López Aguirre and Jérôme Euzenat and Ondřej Šváb-Zamazal and Ian Horrocks and Cássia Trojahn dos Santos},
  title = {{Results of the second evaluation of matching tools}},
  booktitle = {HAL},
  year = {2012},
}
Figure