TL;DR
Standard NLP preprocessing pipelines—like stemming and stop-word removal—can actually hurt the accuracy of Ontology Matching (OM) by creating "over-matches." This paper demonstrates that while basic cleaning helps, aggressive text processing often introduces false positives. To fix this, the authors propose two "repair" strategies: a logic-based approach that protects distinguishing words and a post-hoc LLM-based validator that filters out counter-intuitive matches.
What Problem Does This Paper Address?
Ontology Matching (OM) is the process of finding correspondences between different knowledge graphs (e.g., realizing that "reviews" in one database is the same as "isReviewing" in another). To do this, researchers typically use a standard text preprocessing pipeline: Tokenization, Normalization, Stop Word Removal, and Stemming/Lemmatization.
The problem is that these pipelines were designed for natural language (like news articles), not formal ontologies. In an ontology, a "stop word" like "is" or "not" might carry critical logical meaning. Aggressive stemming can turn two distinct properties into the same root word, leading to a flood of False Positives (FPs). The authors identify a lack of standardization and a lack of understanding regarding how each step in this pipeline actually impacts matching performance.
Methods
The authors divide the pipeline into two phases:
- Phase 1 (T, N): Tokenization and Normalization (e.g., lowercasing).
- Phase 2 (R, S/L): Stop Word Removal and Stemming/Lemmatization.
They found Phase 2 is where the trouble starts. To mitigate this, they introduced two novel repair mechanisms:
- Ad Hoc Logic-Based Repair (AR): This happens before preprocessing. The algorithm scans the ontology for entities that would become identical if processed (e.g., "has_member" and "was_a_member"). It identifies the words that keep them unique ("has" vs "was") and marks them as "reserved," preventing the pipeline from removing or altering them.
- Post Hoc LLM-Based Repair (PR): This happens after preprocessing. The system uses Large Language Models (LLMs) like GPT-4 or Claude as a "sanity check." It prompts the LLM to decide if a match found by the pipeline is actually logically sound, leveraging the LLM’s vast background knowledge to prune nonsensical correspondences.
Datasets
The study evaluated these methods using 8 tracks from the Ontology Alignment Evaluation Initiative (OAEI), encompassing 49 distinct alignments. These tracks cover diverse domains, including:
- Anatomy (Human and Mouse)
- Largebio (Biomedical ontologies like FMA, NCI, and SNOMED)
- Biodiv (Biodiversity and Ecology)
- Conference (Academic organization)
- Food, MSE (Materials Science), and Phenotype.
Key Insights and Ideas
- Ontologies are not "Natural" Language: In standard NLP, "and," "or," and "not" are noise. In ontologies, they are logical operators. Removing them can fundamentally change the meaning of an entity.
- The Precision/Recall Trap: Phase 2 steps (Stemming/Stop words) almost always increase Recall (finding more potential matches) but at a devastating cost to Precision (finding the right matches).
- LLMs as "Detectors," not "Discoverers": Interestingly, the authors found that LLMs are much better at identifying False Positives (wrong matches) than they are at discovering new True Positives from scratch. This makes them ideal for the "repair" phase rather than the initial matching phase.
- Uniqueness as a Constraint: The logic-based repair (AR) is clever because it uses the ontology's own internal requirement—that entity names must be unique—to decide which words are too important to delete.
Key Results
- Phase 1 is a win: Tokenization and Normalization consistently improved the F1-score across nearly all datasets.
- Phase 2 is a gamble: Without repair, stemming and stop-word removal often decreased the overall F1-score because the increase in false matches outweighed the discovery of new ones.
- Repair works: The logic-based repair (AR) significantly improved matching correctness.
- LLM Validation is powerful: Using LLMs as post-hoc validators (PR) reached a "discovery rate" for false matches of over 80% for top-tier models like GPT-4o and Claude 3.
- The "Best" Pipeline: Combining traditional Phase 1 processing with these new repair strategies yielded the highest precision and overall matching performance.
Limitations and Future Work
- Equivalence Only: The study focused strictly on "equivalence" mappings (A = B). It did not address "subsumption" (A is a type of B), which is much harder to validate.
- Gold Standard Issues: The authors noted that even the OAEI "gold standard" references aren't perfect, which can skew accuracy metrics.
- Cost and Latency: While LLM repair is effective, it is significantly more expensive and slower than simple algorithmic logic.
- Future Directions: The authors plan to extend this work to handle complex class axioms and explore how these pipelines interact with knowledge-based OM systems.
Summary updated