Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains

Yang Wu, Raha Moraffah, Rujing Yao, J. H. Yu, Tao Zhang, Xiaozhong Liu2025 · DOI

Abstract

Large Language Models (LLMs) have demonstrated an impressive level of general knowledge. However, they often struggle in highly specialized and cost-sensitive domains such as drug discovery and rare disease research due to the lack of expert knowledge. In this paper, we propose a novel framework (PU-ADKA) designed to efficiently enhance domain-specific LLMs by actively engaging domain experts within a fixed budget. Unlike traditional finetuning approaches, PU-ADKA selectively identifies and queries the most appropriate expert from a team, taking into account each expert's availability, knowledge boundaries, and consultation costs. We train PU-ADKA using simulations on PubMed data and validate it through both controlled expert interactions and real-world deployment with a drug development team, demonstrating its effectiveness in enhancing LLM performance in specialized domains under strict budget constraints. In addition to outlining our methodological innovations and experimental results, we introduce a new benchmark dataset, CKAD, for cost-effective LLM domain knowledge acquisition to foster further research in this challenging area.

Introduction

Large Language Models (LLMs) are impressive generalists, but they often hallucinate or fail when faced with highly specialized, rapidly evolving fields like oncology or rare disease research. Standard solutions like Reinforcement Learning from Human Feedback (RLHF) typically rely on large pools of relatively cheap annotators. However, in sensitive domains, "cheap" feedback doesn't exist—you need specialists whose time is both expensive and scarce.

Furthermore, domain knowledge is often fragmented. One scientist might be an expert in mRNA vaccines but know little about CAR-T cell therapy. Current fine-tuning approaches struggle to identify knowledge boundaries, optimize for cost when experts have different hourly rates, and integrate knowledge discovered after the LLM's training cutoff.

Methodology: The PU-ADKA Framework

The authors introduce the Positive Unlabeled Active Domain Knowledge Acquisition (PU-ADKA) framework, which consists of two primary engines designed to navigate the trade-off between expert cost and knowledge gain.

The PU-ADKA framework architecture, illustrating the simulation environment, PU learning for expert-question matching, and the multi-agent reinforcement learning loop.

1. Expert Allocation via PU Learning

In a real-world scenario, we know an expert is qualified to answer questions about their own published papers (Positive labels). However, we don't know if they can't answer a question just because they didn't write the paper (Unlabeled). PU-ADKA uses Positive-Unlabeled (PU) learning to model these "knowledge boundaries," allowing the system to predict an expert's suitability for a question even without explicit negative data. This acknowledges that expertise is fluid; a cancer researcher might be able to answer a sepsis question if the underlying biological mechanism is similar.

2. Multi-Agent Reinforcement Learning (MARL)

The system treats question selection as a game where multiple agents (representing different experts) compete and cooperate. The goal is to select a set of (question, expert) pairs that maximizes the LLM's performance improvement and data diversity while staying under a strict budget (e.g., $100).

  • Competition: Multiple experts might want the same high-value question; the system assigns it to the one with the highest predicted suitability.
  • Cooperation: The reward function encourages selecting diverse questions to ensure broad knowledge coverage rather than redundant information.

Experimental Setup and Datasets

The authors introduced a new benchmark called CKAD (Cost-effective Knowledge Acquisition Dataset) to evaluate the framework:

  • Source: 48,219 QA pairs extracted from 2024 PubMed Central articles.
  • Temporal Separation: They used Llama2-7B (with a 2023 cutoff) as the base model. This ensures the model cannot answer the questions initially, creating a clean environment to measure new knowledge acquisition.
  • Expert Simulation: They used the top 20 authors in the field as "proxy experts," assigning them costs based on their publication impact factors.
  • Human Validation: They also conducted a real-world deployment with a drug development team consisting of a medical doctor and four PhD researchers.

Key Results

PU-ADKA demonstrated significant improvements over traditional data selection methods across various budget levels.

Performance comparison showing the Win Rate (%) of PU-ADKA against various baselines as the annotation budget increases.
  • Superior Performance: On the CKAD dataset, PU-ADKA achieved a Win Rate (WR) of 18.2% (judged by GPT-4o), significantly higher than the next best baseline, DEITA (14.2%), and nearly triple the random baseline (4.7%).
  • Length-Controlled Success: Even when adjusting for the tendency of LLMs to prefer longer answers (LC_WR), PU-ADKA remained the top performer at 25.6%.
  • Human-In-The-Loop: In the real-world trial with five biomedical experts, PU-ADKA outperformed the best baseline (LESS) by a margin of 2.7% in Win Rate, proving its utility in practical, budget-constrained settings.
  • Budget Efficiency: The model's performance gains were most dramatic at the $100 mark, showing it is particularly effective at rapid knowledge acquisition under tight constraints.

Discussion and Insights

The study reveals that strategic spending is more critical than raw data volume. The model learns that it doesn't always need the most expensive expert for every question. It optimizes the budget by assigning simpler or overlapping questions to junior scholars while saving the budget for senior experts on high-impact, complex queries. Furthermore, the results showed that simply getting more answers (a "Cost-Greedy" approach) is less effective than getting diverse and high-quality answers; PU-ADKA achieved better results with 632 high-quality pairs than a cost-greedy baseline did with 1,000 pairs.

While highly successful in the biomedical field, future work will focus on scaling the MARL coordination for larger expert pools and testing the framework in other high-cost domains like law or finance.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated an impressive level of general knowledge.However, they often struggle in highly specialized and cost-sensitive domains such as drug discovery and rare disease research due to the lack of expert knowledge.In this paper, we propose a novel framework (PU-ADKA) designed to efficiently enhance domain-specific LLMs by actively engaging domain experts within a fixed budget.Unlike traditional finetuning approaches, PU-ADKA selectively identifies and queries the most appropriate expert from a team, taking into account each expert's availability, knowledge boundaries, and consultation costs.We train PU-ADKA using simulations on PubMed data and validate it through both controlled expert interactions and realworld deployment with a drug development team, demonstrating its effectiveness in enhancing LLM performance in specialized domains under strict budget constraints.In addition to outlining our methodological innovations and experimental results, we introduce a new benchmark dataset, CKAD, for cost-effective LLM domain knowledge acquisition to foster further research in this challenging area.
BibTeX
@misc{wu2025active,
  author = {Yang Wu and Raha Moraffah and Rujing Yao and J. H. Yu and Tao Zhang and Xiaozhong Liu},
  title = {{Active Domain Knowledge Acquisition with 100-Dollar Budget: Enhancing LLMs via Cost-Efficient, Expert-Involved Interaction in Sensitive Domains}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.713},
}
Figure