Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answering

Bolei He, Xinran He, Run Shao, Shanfu Shu, Xianwei Xue, Meina Cheng, H. Li, Zhen-Hua Ling2025 · DOI

Abstract

Large Language Models (LLMs) often contain latent domain knowledge from pretraining that remains underutilized in the "long tail" of their probability distributions. We present Select2Know (S2K), a framework that intelligently fuses internal parametric knowledge with external documents at the token level. By focusing on "unmastered" knowledge and employing reasoning-enhanced training, S2K enables models to match the performance of massive domain-specific models while requiring up to 1,000x less training data.

Introduction

Adapting LLMs to specialized domains like medicine, law, or finance typically involves a trade-off between Retrieval-Augmented Generation (RAG) and Continued Pretraining (CPT). RAG is flexible but prone to hallucinations from noisy retrieval, while CPT is computationally expensive, requiring billions of tokens. S2K addresses this by identifying that LLMs often possess "unmastered" or partially internalized knowledge. The framework treats domain adaptation as a two-step process: mastering core concepts through internal-external selection and then applying them via complex reasoning.

Methodology

The S2K framework consists of three primary stages designed for cost-effective post-training.

Overview of the S2K framework, including knowledge fusion, selective SFT, and reasoning-enhanced training.
  1. Internal-External Knowledge Self-Selection: The model performs parallel inferences—one with an external document and one without. By comparing token-level probabilities, the model selects the "internal" token if it is more confident on its own; otherwise, it adopts the "external" token. This creates a high-quality "fusion" dataset.
  2. Selective Supervised Fine-Tuning (Selective SFT): Instead of uniform weighting, Selective SFT uses token-level entropy to identify unfamiliar domains. Higher loss weights are assigned to these "unmastered" tokens, focusing training on new information.
  3. Reasoning-Enhanced Training: To improve problem-solving, S2K generates deductive, inductive, and case-based reasoning data. It then applies Group Relative Policy Optimization (GRPO) to refine the model's reasoning capabilities.

Experimental Evaluation

S2K was evaluated across three high-stakes domains: Medicine (MedQA), Law (JEC-QA), and Finance (FinanceIQ). Training data was synthesized from raw domain corpora rather than human-annotated pairs.

Performance comparison and training token efficiency of S2K against specialized domain models.

Key Results

Experimental results demonstrate that S2K significantly outperforms standard RAG and few-shot prompting. Notably, S2K matches or exceeds the performance of specialized models like BioMistral and Saul-7B. Most importantly, S2K achieves this with a massive reduction in data requirements. For instance, in the medical domain, S2K reached competitive performance using only 0.04 billion tokens, compared to the 3 billion tokens required by BioMistral—representing a 2-to-3 order of magnitude reduction in training data and associated compute costs.

Discussion and Conclusion

S2K proves that "more data" is not always the answer; rather, focusing on the model's specific knowledge gaps is more efficient. While currently focused on static domain knowledge, future work will explore integrating RAG for real-time updates and scaling the framework to larger architectures.

Summary updated

BibTeX
@misc{he2025select,
  author = {Bolei He and Xinran He and Run Shao and Shanfu Shu and Xianwei Xue and Meina Cheng and H. Li and Zhen-Hua Ling},
  title = {{Select to Know: An Internal-External Knowledge Self-Selection Framework for Domain-Specific Question Answering}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.565},
}
Figure