Symmetry- and Asymmetry-Aware Dual-Path Retrieval and In-Context Learning-Based LLM for Equipment Relation Extraction

Mingfei Tang, Liang Zhang, Zhipeng Yu, Xiaolong Shi, Xiulei LiuSymmetry 2025 · DOI

TL;DR

This paper introduces a novel approach for extracting relationships between equipment entities, tackling the challenge of complex and overlapping relationships often found in this domain. By combining domain-specific fine-tuning of language models with a dual-path retrieval strategy and in-context learning, the method significantly improves extraction accuracy, outperforming existing state-of-the-art models.

What Problem Does This Paper Address?

Relation extraction (RE) aims to identify semantic relationships between entities in text and represent them as structured triples (head entity, relation, tail entity). While RE is crucial for knowledge graph construction and information retrieval, the equipment domain presents unique challenges. These include:

  • Asymmetric Patterns: Entities often participate in multiple overlapping relations, breaking the expected symmetry of semantic associations.
  • Overlapping Relations: This can manifest as Single Entity Overlap (SEO), where one entity has multiple relations with different entities, or Entity Pair Overlap (EPO), where multiple relations exist between the same entity pair.
  • Domain-Specific Terminology: The equipment domain uses highly specialized language, complex technical descriptions, and unique semantic relationships that general-purpose models struggle to capture.

These challenges lead to reduced accuracy in conventional RE methods. This paper aims to address these issues by developing a model that can effectively handle overlapping relations and leverage domain-specific knowledge.

Methods

The proposed method, "Symmetry- and Asymmetry-Aware Dual-Path Retrieval and In-Context Learning-Based LLM," comprises four key components:

  1. Embedding Model Fine-Tuning Module: The pre-trained BGE-M3 multilingual embedding model is fine-tuned on a domain-specific equipment corpus. This adaptation enhances its ability to capture nuanced semantics and terminology relevant to the equipment domain, providing higher-quality semantic vectors. The lower layers of BGE-M3 are frozen, and only a lightweight domain adaptation layer is updated.
  2. Retrieval Example Library Construction Module: A repository of demonstration examples is built. Each example includes the text, sentence-level and relation-level embeddings (generated by the fine-tuned BGE-M3), and detailed reasoning logic. This reasoning logic is generated using a structured template that guides a large language model (LLM) to explain the inference process for each example.
  3. Relevance Computation Module: This module implements a novel dual-path retrieval strategy:
  4. Sentence-Level Embedding Retrieval: Captures global semantic information by comparing the input sentence's embedding with those in the repository.
  5. Relation-Level Embedding Retrieval: Focuses on local semantic information by concatenating subject and object entity embeddings to represent the relation. The results from both paths are fused and re-ranked using Reciprocal Rank Fusion (RRF) to improve retrieval coverage and accuracy.

  6. Prompt Construction Module: Task-specific prompt templates are constructed. These templates combine instructions, the retrieved demonstration examples (ordered by relevance), and the input text. This structured prompt is then fed to an LLM (GPT-4) to perform relation extraction in a contextual learning setting.

Datasets

The experiments were conducted on a self-annotated equipment relation dataset. This dataset was constructed by annotating knowledge data from Wikipedia. The data was categorized into three overlap types: Normal, SEO, and EPO. A total of 8457 data entries were annotated and divided into a test set (6766 sentences) and a retrieval example set (1691 sentences) at an 8:2 ratio.

Key Insights and Ideas

  • Domain Adaptation is Crucial: General-purpose embedding models struggle with specialized domains. Fine-tuning on domain-specific data significantly improves the quality of semantic representations for retrieval.
  • Dual-Path Retrieval Captures Complementary Information: Combining sentence-level (global context) and relation-level (local entity interactions) retrieval provides a more comprehensive understanding of relationships, especially in complex scenarios.
  • Reasoning Logic Enhances In-Context Learning: Providing explicit reasoning steps for demonstration examples helps LLMs better understand the task and generalize to new instances.
  • RRF for Robust Fusion: Reciprocal Rank Fusion effectively combines rankings from different retrieval strategies, leading to more reliable and accurate example selection.
  • EPO Remains a Challenge: While the model performs well on normal and SEO cases, Entity Pair Overlap (EPO) scenarios, where multiple relations exist between the same entity pair, are still the most challenging. This highlights the difficulty in disambiguating semantically close relations and the limitations of current reasoning generalization.

Key Results

The proposed model achieved an F1-score of 88.53% on the equipment domain dataset, which is an improvement of 2.48% in Precision, 2.13% in Recall, and 2.32% in F1-score over the best-performing baseline, GPT-RE.

  • The optimal number of retrieved examples (k) was found to be 5, yielding an F1-score of 88.53%. Increasing k beyond 5 led to performance degradation due to noise and redundancy.
  • The model performed best on "Normal" relation types (F1: 90.12%), followed by "SEO" (F1: 88.60%), and then "EPO" (F1: 86.87%), indicating EPO as the most challenging scenario.
  • Ablation studies showed that removing the domain adaptation layer resulted in a 2.63% drop in F1-score, highlighting its importance. The dual-path retrieval strategy (Sentence + Relation) achieved an F1-score of 88.53%, outperforming single-path retrieval methods by 1.73%.

Limitations and Future Work

  • Scalability and Generalizability: The current work focuses on intra-sentence extraction and is tailored to the equipment domain. Future work should explore document-level RE and test the model's adaptability to other domains and languages.
  • Entity Pair Overlap (EPO) Handling: The model's performance on EPO remains suboptimal due to limitations in relation-level retrieval, semantic interference in multi-relation learning, and reasoning logic generalization. Future research should focus on enhanced retrieval mechanisms and more sophisticated reasoning for EPO.
  • Qualitative Analysis of Failure Cases: A deeper dive into failure cases, particularly for EPO, is needed to inform the design of more robust algorithms and reasoning templates.
  • Broader Applications: The authors suggest potential applications beyond the military domain, such as knowledge graph construction in civilian and industrial sectors.

Summary updated

BibTeX
@article{tang2025symmetry,
  author = {Mingfei Tang and Liang Zhang and Zhipeng Yu and Xiaolong Shi and Xiulei Liu},
  title = {{Symmetry- and Asymmetry-Aware Dual-Path Retrieval and In-Context Learning-Based LLM for Equipment Relation Extraction}},
  journal = {Symmetry},
  year = {2025},
  doi = {10.3390/sym17101647},
}
Figure