UniHGKR: Unified Instruction-aware Heterogeneous Knowledge Retrievers

Dehai Min, Zhiyang Xu, Guilin Qi, Lifu Huang, Chenyu You

Abstract

Existing information retrieval (IR) models often assume a homogeneous structure for knowledge sources and user queries, limiting their applicability in real-world settings where retrieval is inherently heterogeneous and diverse. In this paper, we introduce UniHGKR, a unified instruction-aware heterogeneous knowledge retriever that (1) builds a unified retrieval space for heterogeneous knowledge and (2) follows diverse user instructions to retrieve knowledge of specified types. UniHGKR consists of three principal stages: heterogeneous self-supervised pretraining, text-anchored embedding alignment, and instruction-aware retriever fine-tuning, enabling it to generalize across varied retrieval contexts. This framework is highly scalable, with a BERT-based version and a UniHGKR-7B version trained on large language models. Also, we introduce CompMix-IR, the first native heterogeneous knowledge retrieval benchmark. It includes two retrieval scenarios with various instructions, over 9,400 question-answer (QA) pairs, and a corpus of 10 million entries, covering four different types of data. Extensive experiments show that UniHGKR consistently outperforms state-of-the-art methods on CompMix-IR, achieving up to 6.36% and 54.23% relative improvements in two scenarios, respectively. Finally, by equipping our retriever for open-domain heterogeneous QA systems, we achieve a new state-of-the-art result on the popular ConvMix task, with an absolute improvement of up to 5.90 points.

Introduction: The Challenge of Heterogeneous Retrieval

Modern Information Retrieval (IR) and Retrieval-Augmented Generation (RAG) systems are typically optimized for homogeneous data, primarily plain text. However, real-world knowledge is often fragmented across diverse formats, including Knowledge Graphs (KG), structured tables, and infoboxes. Conventional systems handle this by maintaining separate indices for each data type or by converting all data to text, which often results in the loss of structural nuances and increased computational overhead.

Comparison between conventional retrievers and the unified UniHGKR approach.

Furthermore, most retrievers are "instruction-blind," lacking the ability to prioritize specific sources based on natural language requests. UniHGKR addresses these limitations by creating a unified embedding space where heterogeneous data types are aligned and can be retrieved based on specific user instructions.

Methodology: A Three-Stage Training Pipeline

UniHGKR utilizes a sophisticated three-stage training process to achieve instruction-aware, unified retrieval.

The three-stage training pipeline of UniHGKR: pretraining, alignment, and fine-tuning.
  1. Unified Embedding Self-Supervised Pre-training: The model is pretrained on data-text pairs, such as a linearized table row paired with a descriptive sentence. Using a token-masking reconstruction task, the model learns to understand the underlying structure of heterogeneous data.
  2. Text-Anchored Heterogeneous Embedding Alignment: Contrastive learning is employed to align structured data embeddings (e.g., KG triples) with their natural language descriptions. This "anchors" all data types to a common semantic space, ensuring that semantically related information across different formats is mapped to nearby vectors.
  3. Instruction-Aware Fine-tuning: The core innovation involves training the model to follow specific retrieval instructions. Two specialized contrastive losses are introduced: a type-balanced loss to ensure fair treatment across data types when no source is specified, and a type-preferred loss to prioritize requested sources by adjusting negative sampling during training.

The framework is highly scalable, implemented as both a BERT-based model (UniHGKR-base) and a 7B parameter version (UniHGKR-7B) based on the LLARA architecture. A key design insight was that keeping structured data intact through linearization, rather than fragmenting it, prevents the loss of vital semantic context.

Evaluation and Benchmarking

To evaluate UniHGKR, the authors introduced CompMix-IR, the first native heterogeneous knowledge retrieval benchmark. It features over 9,400 QA pairs and a corpus of 10 million entries across four data types: Text (57.7%), KG (21.6%), Tables (10.2%), and Infoboxes (10.5%).

Key Results

  • State-of-the-Art Performance: UniHGKR-base outperformed significantly larger models, while UniHGKR-7B set new records on CompMix-IR.
  • Instruction Following: In scenarios requiring specific data types, UniHGKR achieved a 54.23% relative improvement in table retrieval compared to top baselines.
  • Downstream Impact: Integrating UniHGKR into an open-domain QA system for the ConvMix task resulted in a 5.90-point absolute improvement in Precision@1.
  • Zero-Shot Generalization: The model demonstrated robust performance on the BEIR benchmark, outperforming specialized models like SimCSE and DiffCSE in unseen domains.

Conclusion and Future Directions

UniHGKR demonstrates that a unified, instruction-aware approach can significantly enhance retrieval performance across heterogeneous data sources. By treating text as a universal anchor and incorporating instruction-awareness, the model moves beyond simple relevance to understand user intent regarding data sources.

Future work will focus on expanding the model's capabilities to broader domains beyond the current five (books, movies, music, TV, and football), handling more complex multi-source instructions, and incorporating additional modalities such as images and audio into the unified retrieval space.

Summary updated

Abstract

Existing information retrieval (IR) models often assume a homogeneous structure for knowledge sources and user queries, limiting their applicability in real-world settings where retrieval is inherently heterogeneous and diverse. In this paper, we introduce UniHGKR, a unified instruction-aware heterogeneous knowledge retriever that (1) builds a unified retrieval space for heterogeneous knowledge and (2) follows diverse user instructions to retrieve knowledge of specified types. UniHGKR consists of three principal stages: heterogeneous self-supervised pretraining, text-anchored embedding alignment, and instruction-aware retriever fine-tuning, enabling it to generalize across varied retrieval contexts. This framework is highly scalable, with a BERT-based version and a UniHGKR-7B version trained on large language models. Also, we introduce CompMix-IR, the first native heterogeneous knowledge retrieval benchmark. It includes two retrieval scenarios with various instructions, over 9,400 question-answer (QA) pairs, and a corpus of 10 million entries, covering four different types of data. Extensive experiments show that UniHGKR consistently outperforms state-of-the-art methods on CompMix-IR, achieving up to 6.36% and 54.23% relative improvements in two scenarios, respectively. Finally, by equipping our retriever for open-domain heterogeneous QA systems, we achieve a new state-of-the-art result on the popular ConvMix task, with an absolute improvement of up to 5.90 points.
BibTeX
@inproceedings{min2025unihgkr,
  author = {Dehai Min and Zhiyang Xu and Guilin Qi and Lifu Huang and Chenyu You},
  title = {{UniHGKR: Unified Instruction-aware Heterogeneous Knowledge Retrievers}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=nKzPKwIZBz},
}

Loading related content...

Figure