Implementing Semantic Precision and Recall

Daniel Fleischhacker, Heiner StuckenschmidtOM 2009 · OpenReview

Abstract

Traditional evaluation metrics for ontology alignment, such as precision and recall, often fail to account for the underlying logic of the ontologies involved. This paper presents a practical implementation of "restricted" semantic precision and recall. By evaluating alignments based on their deductive consequences rather than simple set-to-set comparisons, this approach provides a more accurate assessment of alignment quality. The authors demonstrate the utility of these metrics using results from the 2008 OAEI campaign, showing significant shifts in performance metrics when semantics are considered.

Introduction

In the Semantic Web, ontology alignment is the critical process of identifying correspondences between entities in disparate ontologies—for instance, mapping "Staff" in one schema to "Employee" in another. Historically, the performance of alignment algorithms has been measured using standard Information Retrieval (IR) metrics: Precision and Recall.

However, these traditional metrics are "blind" to semantics. They treat alignments as simple lists of strings or identifiers. If an algorithm identifies a match that is logically implied by the ontology but not explicitly listed in the "gold standard" reference, traditional metrics penalize it as a mistake. While theoretical frameworks for semantic precision and recall have been proposed, they have often been too computationally complex or theoretically inconsistent for practical use. This paper bridges that gap by introducing a restricted, computable version of these metrics.

Methods: Restricted Semantic Precision and Recall

The core innovation is the transition from comparing sets of correspondences to comparing their deductive closures. The authors define Restricted Semantic Precision and Recall through several key components:

  1. Aligned Ontology: The two ontologies being matched, combined with the proposed alignment, are treated as a single logical system.
  2. $\alpha$-consequences: These represent correspondences that are not explicitly stated but are logically implied by the combined system.
  3. Closure ($Cn$): This is the set of all possible logical consequences derived from an alignment.
  4. The Metric: Instead of a direct comparison between Alignment $A$ and Reference $R$, the metrics compare the closure of $A$ to the closure of $R$.

To ensure computational feasibility and avoid theoretical edge cases, the scope is restricted to non-complex alignments involving single, atomic concepts. The authors also employ "natural" and "pragmatic" semantics to convert correspondences into formal axioms.

Experimental Setup

The implementation was evaluated using data from the 2008 Ontology Alignment Evaluation Initiative (OAEI). The study focused on the "conference" test set, which involves matching ontologies related to academic conference organization. The metrics were applied to the outputs of three prominent matching systems: ASMOV, DSSim, and Lily.

Results and Analysis

The experiments demonstrated that incorporating semantics significantly alters the perceived quality of a matcher. Semantic measures generally reward systems for implicit truths uncovered through deductive closure.

Comparison of classical and semantic precision/recall across different matchers and thresholds.

As shown in the table above, the shift from classical to semantic measures is pronounced:

  • ASMOV: At a confidence threshold of 0.2, classical precision and recall were both 0.42. Utilizing "pragmatic" semantic measures, precision increased to 0.49, while recall saw a substantial jump to 0.74.
  • DSSim: This matcher exhibited an even more dramatic shift in recall, rising from 0.52 (classical) to 0.88 (semantic) at the 0.2 threshold.
  • Lily: Similar trends were observed, with semantic measures providing a more nuanced view of the matcher's logical consistency.

Key Insights

  • Logic Over Labels: A match is a logical claim, not just a pair of labels. If an alignment asserts $A = B$ and the ontology defines $B$ as a subclass of $C$, the alignment implicitly claims $A$ is a subclass of $C$. Semantic measures correctly reward these logical implications.
  • Recall Boost: Semantic measures almost universally result in higher recall scores because the deductive closure identifies correct correspondences that were logically present but omitted from explicit reference lists.
  • Threshold Sensitivity: Semantic measures proved more robust across different confidence thresholds than traditional measures, offering a more stable assessment of a system's logical performance.

Discussion and Future Work

A primary limitation noted by the authors is the current state of the field: most existing matchers and reference datasets are limited to simple equivalence mappings. Consequently, the full potential of semantic measures is not yet fully realized.

The authors suggest that as the field evolves toward complex mappings—specifically subsumption correspondences (hierarchical relationships)—semantic measures will become indispensable. Future research will focus on extending reference alignments to include these hierarchical relationships and refining the reasoning engines used to calculate deductive closures.

Summary updated

BibTeX
@inproceedings{fleischhacker2009implementing,
  author = {Daniel Fleischhacker and Heiner Stuckenschmidt},
  title = {{Implementing Semantic Precision and Recall}},
  booktitle = {OM},
  year = {2009},
  url = {https://openreview.net/forum?id=injfKky_UYn},
}
Figure