Description of alignment evaluation and benchmarking results

Pavel Shvaiko, Jérôme Euzenat, Heiner Stuckenschmidt, Malgorzata Mochól, Fausto Giunchiglia, Mikalai Yatskevich, Paolo Avesani, Willem Robert van Hage, Ondřej Šváb, Vojtěch SvátekHAL 2007

TL;DR

This paper introduces new semantic measures for evaluating ontology alignment, moving beyond simple syntactic comparisons. It then presents the results of the OAEI-2006 campaign, showcasing significant advancements in ontology matching technology and highlighting the growing maturity of the field.

What Problem Does This Paper Address?

Ontology matching is crucial for integrating information from diverse sources, but evaluating the quality of these matches has been a persistent challenge. Traditional precision and recall measures, borrowed from information retrieval, treat alignments as simple sets of correspondences, failing to account for semantic nuances. This can lead to situations where semantically equivalent alignments are penalized, or where slightly incorrect but semantically close matches are indistinguishable from completely wrong ones. The paper addresses this by proposing semantically grounded evaluation measures and by reporting on the OAEI-2006 campaign, a large-scale effort to benchmark ontology matching systems.

Methods

The paper introduces novel semantic precision and recall measures. These measures go beyond syntactic overlap by considering the consequences of alignments within the semantic framework of ontologies. This allows for a more nuanced evaluation that accounts for semantic equivalence and proximity.

The paper also details the OAEI-2006 campaign, a coordinated international initiative to evaluate ontology matching systems. This campaign involved:

  • Four tracks: Benchmarks, expressive ontologies, directories and thesauri, and a conference track with a consensus workshop.
  • Six data sets across these tracks.
  • Ten participating systems, an increase from previous years.
  • Standard evaluation measures (precision and recall) augmented by the new semantic measures and precision/recall graphs.

Datasets

The OAEI-2006 campaign utilized several datasets across its tracks:

  • Benchmark track: A systematic benchmark test set based on a reference ontology for bibliographic references, with numerous variations.
  • Expressive ontologies track:
    • Anatomy: Large real-world ontologies (over 50MBs) from the Foundational Model of Anatomy (FMA) and Galen.
  • Jobs: Real-world business case data from the employment sector (OWL ontologies).
  • Directories and thesauri track:
    • Directory: Matching web directories (Google, Yahoo, Looksmart) with a large number of elementary tests.
  • Food: Two SKOS thesauri (AGROVOC and NALT) about food.
  • Conference track and consensus workshop: A collection of conference organization ontologies.

The paper also discusses the creation of new datasets for evaluating recall and precision in the directory and food tracks, respectively, using semi-automatic and manual annotation methods.

Key Insights and Ideas

  • Semantic evaluation is crucial: Moving beyond syntactic comparisons is essential for accurately assessing ontology alignment quality, especially as ontologies become more complex.
  • OAEI is driving progress: The Ontology Alignment Evaluation Initiative (OAEI) is a vital platform for advancing the field, fostering competition and collaboration, and pushing the boundaries of matching technology.
  • Matching technology is maturing: The OAEI-2006 results show significant improvements in system robustness, scalability, and the ability to handle complex real-world data, suggesting that matching technology is becoming ready for large-scale applications.
  • Challenges remain: Despite progress, significant challenges persist, particularly in handling irregular concept names, leveraging semantic reasoning, and dealing with the inherent ambiguity and context-dependency of ontologies.
  • Consensus building is valuable: Workshops and manual validation are crucial for understanding the nuances of alignment quality and identifying subtle errors that automated measures might miss.

Key Results

  • Improved Semantic Measures: The proposed semantic precision and recall measures were shown to be more discriminating than traditional syntactic measures, particularly in the motivating example.
  • OAEI-2006 Participation: The campaign saw an increase in participants (10 systems) and a general improvement in results compared to previous years.
  • Benchmark Track: Three systems (Falcon, COMA++, and RiMOM) competed at the highest level, achieving results comparable to the best system in 2005.
  • Expressive Ontologies Track: For the first time, several systems were able to submit results for the large anatomy dataset, indicating advancements in handling scalability.
  • Directory Track: Systems generally showed better results than in OAEI-2005, with Falcon demonstrating a significant increase in approximate recall.
  • Food Track: A "high agreement" was observed between the best three systems (RiMOM, Falcon, and HMatch), indicating good performance on this thesauri matching task.
  • Conference Track: The consensus workshop revealed insights into the importance of context, semantic relations, and alternative interpretations for accurate alignment.

Limitations and Future Work

  • Limitations:
    • The benchmark test case was found to be not discriminant enough between systems.
  • The absence of a reference alignment in some cases could not be fully compensated by cross-validation.
  • The time allocated for evaluation was often insufficient.
  • The semantic measures require a prover for ontology languages, which can be computationally intensive.
  • Some systems struggled with irregular concept names and the "use" of concepts modeled via skos:altLabel.
  • Future Work:
    • Implement and apply the semantic precision and recall measures to larger datasets, including OAEI campaign results.
  • Continue running the OAEI campaign, focusing on improving test cases, testing methodology, and accepting continuous submissions.
  • Explore new real-world test cases and establish general rules for consensus reference alignment acquisition and application-oriented evaluation.
  • Investigate better alternatives for the benchmark test case.
  • Explore gradual degradation of benchmark tests for more realistic evaluation.
  • Allocate more time for future evaluation campaigns.
  • Develop matching tools capable of computing subsumption relations, not just equivalences.
  • Further investigate the use of logical reasoning and context in ontology matching.

Summary updated

Abstract

shvaiko2007a
BibTeX
@inproceedings{shvaiko2007description,
  author = {Pavel Shvaiko and Jérôme Euzenat and Heiner Stuckenschmidt and Malgorzata Mochól and Fausto Giunchiglia and Mikalai Yatskevich and Paolo Avesani and Willem Robert van Hage and Ondřej Šváb and Vojtěch Svátek},
  title = {{Description of alignment evaluation and benchmarking results}},
  booktitle = {HAL},
  year = {2007},
}
Figure