KBAlign: Efficient Self Adaptation on Specific Textual Knowledge Bases

Zheni Zeng, Yuxuan Chen, Yu Shi, Ruobing Wang, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun

Abstract

Retrieval-Augmented Generation (RAG) is essential for grounding Large Language Models (LLMs) in factual data, yet adapting these models to specialized, small-scale knowledge bases (KBs) remains a challenge. Current adaptation methods often rely on expensive human annotations or synthetic data from proprietary "teacher" models like GPT-4. We present KBAlign, a self-supervised framework that allows LLMs to autonomously align with specific KBs. By utilizing multi-grained self-annotation and iterative self-verification, KBAlign achieves approximately 90% of the performance gains seen with GPT-4-supervised methods while maintaining cost-efficiency and data privacy.

Introduction

While RAG effectively grounds LLMs, performance often degrades in specialized domains—such as legal or medical databases—where general-purpose models struggle with domain-specific terminology and nuances. Existing solutions for domain adaptation typically involve unsupervised training on raw text, which fails to teach the model how to utilize knowledge for question answering (QA), or high-quality fine-tuning, which is prohibitively expensive or poses privacy risks.

Overview of the KBAlign framework showing self-annotation, iterative verification, and targeted inference.

KBAlign addresses these issues by enabling a model to act as its own teacher. It aligns the model's internal representations with a specific KB using only its intrinsic capabilities, bridging the gap between raw text exposure and supervised fine-tuning.

Methodology

The KBAlign framework consists of three primary stages designed to mimic a student studying for an open-book exam:

1. Multi-Grained Self-Annotation

The model generates its own training data by "reading" the KB at different granularities:

  • Short-dependency: The model generates Q&A pairs from single paragraphs to master localized facts.
  • Long-dependency: By identifying related segments across the KB through hierarchical structures or vector similarities, the model generates questions requiring multi-hop reasoning. This strategy specifically targets the "global knowledge" challenge where models often struggle to connect disparate pieces of information.

2. Iterative Tuning

Rather than a single training pass, KBAlign employs a multi-stage "self-correction" loop. The model is initially tuned on a subset of self-generated data. It then attempts to answer new questions and performs self-verification to identify errors. These error analyses serve as refined training signals, significantly accelerating the model's convergence to the domain.

Detailed technical workflow of KBAlign, illustrating the iterative process from initial self-annotation through iterative tuning and final targeted inference.

3. Targeted Inference

During inference, the adapted model utilizes Query Expansion. It leverages its newly acquired domain knowledge to enrich user queries before retrieval. This ensures that the retriever pulls more relevant documents, compensating for the limitations of standard vector-based search.

Evaluation and Results

KBAlign was evaluated across four benchmarks: LooGLE (long-context/local knowledge), ASQA (multi-segment synthesis), JEC-QA (legal examination), and BioASQ (biomedical QA).

Key results include:

  • Performance Parity: KBAlign reached 90% of the performance gains of models supervised by GPT-4, despite using much smaller annotator models (e.g., 2B or 8B parameters).
  • Significant Accuracy Gains: On the LooGLE benchmark, a MiniCPM-2B model's F1 score improved from 30.92 (Vanilla RAG) to 54.09 using KBAlign.
  • Efficiency: The framework proved more effective and faster than traditional language modeling on raw text. The query expansion strategy in the inference phase was particularly effective at improving retrieval relevance.

Discussion and Insights

A central finding of this work is that even small-scale LLMs possess sufficient latent reasoning to serve as their own supervisors if the annotation process is properly structured. The iterative self-verification phase is critical; by forcing the model to analyze its own mistakes (e.g., recognizing missing information in a multi-part answer), it learns domain boundaries more effectively than through passive reading. Furthermore, the distinction between local and global knowledge mastery highlights that long-dependency reasoning is the primary bottleneck in domain adaptation.

Limitations

Despite its successes, KBAlign faces challenges such as "catastrophic forgetting," where intensive domain tuning slightly reduces general instruction-following capabilities. Additionally, while the generator (LLM) adapts, the retriever remains static, suggesting that future work should explore self-supervised adaptation of the retrieval component itself. Finally, holistic "bird's-eye" analysis of massive corpora remains a complex frontier for this approach.

Summary updated

Abstract

Although retrieval-augmented generation (RAG) remains essential for knowledge-based question answering (KBQA), current paradigms face critical challenges under specific domains. Existing methods struggle with targeted adaptation on small-scale KBs: vanilla unsupervised training exhibits poor effectiveness, while fine-tuning incurs prohibitive costs of external signals. We present KBAlign, a self-supervised framework that enhances RAG systems through efficient model adaptation. Our key insight is to leverage the model's intrinsic capabilities for knowledge alignment through two innovative mechanisms: multi-grained self-annotation that captures global knowledge for data construction, and iterative tuning that accelerates convergence through self verification. This framework enables cost-effective model adaptation to specific textual KBs, without human supervision or external model assistance. Experiments demonstrate that KBAlign can achieve 90\% of the performance gain obtained through GPT-4-supervised adaptation, while relying entirely on self-annotation of much smaller models. KBAlign significantly improves downstream QA accuracy across multiple domains with tiny costs, particularly benefiting scenarios requiring deep knowledge integration from specialized corpora. We release our experimental data, models, and process analyses to the community for further exploration (https://github.com/thunlp/KBAlign).
BibTeX
@misc{zeng2024kbalign,
  author = {Zheni Zeng and Yuxuan Chen and Yu Shi and Ruobing Wang and Yukun Yan and Zhenghao Liu and Shuo Wang and Xu Han and Zhiyuan Liu and Maosong Sun},
  title = {{KBAlign: Efficient Self Adaptation on Specific Textual Knowledge Bases}},
  year = {2024},
  eprint = {2411.14790},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.14790},
}
Figure