Group, Embed and Reason: A Hybrid LLM and Embedding Framework for Semantic Attribute Alignment

Shramona Chakraborty, Shashank Mujumdar, Nitin Gupta, Sameep Mehta, Ronen Kat, Itay Etelis, Mohamed Mahameed, Itai Guez, Rachel Tzoref-BrillEMNLP 2025

TL;DR

This paper introduces a novel, unsupervised framework for automatically aligning semantically equivalent attributes across different data schemas, even when only metadata (names and descriptions) is available. By combining the contextual reasoning of Large Language Models (LLMs) with the stability of embedding-based similarity, the framework achieves scalable and accurate schema matching in privacy-constrained, schema-only settings.

What Problem Does This Paper Address?

In many enterprise systems, data integration tasks like API connections, data consolidation, and ETL pipeline creation rely heavily on accurately matching attributes that represent the same real-world concept but are named or described differently across various data sources. This "semantic attribute alignment" is crucial for unifying data, enabling consistent reporting, and driving accurate analytics. The challenge is amplified when dealing with "schema-only" scenarios, where no actual data instances are available for comparison. This often leads to ambiguous attribute names, inconsistent descriptions, and varying naming conventions, making traditional methods that rely on simple string matching or surface-level comparisons insufficient. Furthermore, privacy concerns in domains like healthcare and finance restrict access to instance-level data, necessitating solutions that work solely with metadata.

Methods

The proposed framework is a hybrid, unsupervised approach designed to overcome the limitations of existing methods. It consists of four key stages:

  1. Semantic Grouping: To address the token limitations of LLMs, attributes within source and target schemas are first clustered into semantically coherent groups. This is achieved by embedding attribute names and descriptions and then applying clustering (k-Nearest-Neighbors) to create groups that fit within the LLM's context window while preserving relatedness.
  2. GenAI-based Mapping Generation: For each group, a carefully designed prompt is fed to an LLM. The LLM is instructed to generate candidate semantic mappings between source and target attributes, along with natural language justifications for each mapping. This leverages the LLM's contextual reasoning capabilities.
  3. Embedding-Based Enhancement and Pruning: This stage acts as a validation and refinement layer.
  4. Enhancement: Cosine similarity is computed between attribute embeddings. If a pair has a high similarity score but was missed by the LLM, it's added as a candidate mapping.
  5. Pruning: If an LLM-generated mapping has a low embedding similarity score, it's removed, helping to eliminate spurious or semantically weak alignments.
  6. Information Processing and Final Mapping Selection: The final stage refines the candidate mappings through:
  7. Hallucination Removal: Mappings where the LLM generated pairs not present in the original schemas are identified and removed.
  8. Justification Pruning: A zero-shot entailment classifier is used to assess the plausibility of the LLM's justifications. Mappings with weak or unconvincing justifications are discarded.
  9. Ranking: For attributes with multiple candidate mappings, an embedding-based ranking mechanism prioritizes those with higher semantic similarity.

Datasets

The paper evaluates its framework on five schema-only datasets from real and synthetic healthcare sources:

  • CMS: A large-scale Medicare schema with minimal metadata.
  • SAKI: Curated interoperability schemas with moderate metadata and semantic ambiguities.
  • MIMIC-III: An ICU dataset with a structured schema and rich metadata.
  • MIMIC-IV: An updated version of MIMIC-III with a restructured schema and new fields.
  • Synthea: A synthetic EHR dataset with clean metadata, used for evaluation under ideal and complex mapping scenarios.

Expert-aligned mappings were used where available for evaluation.

Key Insights and Ideas

  • Hybrid Approach is Key: The paper strongly advocates for combining LLM's contextual reasoning with the stable, semantic similarity of embeddings. Neither approach alone is sufficient for robust schema matching, especially in schema-only settings.
  • LLM Limitations Addressed: The framework directly tackles LLM weaknesses like token limits (via grouping) and hallucinations (via justification pruning and hallucination removal).
  • Unsupervised Power: The entire framework operates without requiring labeled training data or instance-level data, making it highly practical for privacy-sensitive domains and scenarios with limited resources.
  • Layered Filtering for Robustness: The multi-stage filtering (embedding pruning, justification pruning, hallucination removal) is crucial for improving precision and ensuring the reliability of the final mappings.

Key Results

The proposed framework demonstrates consistent improvements over string similarity and embedding-based baselines, achieving F1-scores that are 7% to 40% higher across various datasets and LLM variants.

  • LLM Performance: The DeepSeek family of LLMs showed particularly strong performance, attributed to their instruction-following capabilities and reduced hallucination.
  • Comparison to Supervised: Even without supervised training, the framework achieved F1-scores within 12–18% of SMAT, a supervised model, highlighting its effectiveness in zero-shot settings.
  • Module Contributions: Removing either the Embedding Filtering (EF) or Justification Pruning (JP) module led to a moderate performance decline (2–5%), underscoring the complementary roles of these components. Simultaneous removal of both EF and JP resulted in the largest performance degradation.
  • Qualitative Analysis: The paper highlights instances where the framework correctly identified mappings missed by ground truth, inferred reasonable mappings for missing GT, and effectively filtered out spurious or weak predictions.

Limitations and Future Work

The authors acknowledge several limitations:

  • Metadata Sensitivity: The framework's performance is dependent on the quality of textual metadata. Sparse or ambiguous descriptions can hinder both embedding and LLM performance.
  • Grouping Quality: The effectiveness of the LLM stage is influenced by the quality of semantic grouping. Incoherent groups might lead to suboptimal mappings.
  • Justification Ambiguity: While justification pruning is effective, it might inadvertently filter out valid mappings if the LLM's explanation is weak or generic, especially for domain-specific synonyms or abbreviations.
  • No Learning from Feedback: The current framework does not learn from user feedback or outcomes over time.

Future work could explore more sophisticated grouping mechanisms (e.g., incorporating ontologies), improving justification scoring robustness, and developing lightweight fine-tuning or feedback loops to adapt the system across domains and enhance long-term generalizability.

Summary updated

BibTeX
@inproceedings{chakraborty2025group,
  author = {Shramona Chakraborty and Shashank Mujumdar and Nitin Gupta and Sameep Mehta and Ronen Kat and Itay Etelis and Mohamed Mahameed and Itai Guez and Rachel Tzoref-Brill},
  title = {{Group, Embed and Reason: A Hybrid LLM and Embedding Framework for Semantic Attribute Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure