Abstract
The substantial interest in updating Large Language Models (LLMs) without retraining from scratch is accompanied by several challenges. This is particularly true when updating LLMs with datasets that necessitate domain-expert reasoning across extensive texts, despite limited samples. We termed the scenario as the Few-Shot Domain-Expert Reasoning for Updating LLMs (FDoR-UL). Traditional methods such as Low-Rank Adaptation (LoRA) and Retrieval Augmented Generation (RAG) are inadequate for addressing this critical issue, particularly evident in our exploration of a specific medical dataset that epitomizes the distinct needs of FDoR-UL. To tackle this challenge, we introduce a Sequential Fusion method to integrate knowledge from complex contexts into LLMs. This method employs a two-stage framework: initially leveraging general LLMs to perform relation extraction for knowledge acquisition from complex texts, followed by updating domain-specific LLMs through Knowledge Editing (KE). Employing our method, domain-specific LLMs achieved a 71.7% accuracy (an average gain of 39.1%) in question-answering tasks. Furthermore, we expanded our evaluation to a novel economics-management dataset we developed, where our method achieved a 75.0% accuracy (an average gain of 45.0%). These findings underscore the effectiveness and flexibility of our approach in FDoR-UL across various domains.
Introduction
Updating Large Language Models (LLMs) with specialized, "expert-level" knowledge is notoriously difficult, especially when data is scarce—a scenario defined as Few-Shot Domain-Expert Reasoning for Updating LLMs (FDoR-UL). In fields like medicine or economics, the "truth" is often buried in dense, professional prose. Simply feeding this raw text to a model via traditional methods often results in reasoning errors or hallucinations.
Traditional approaches usually fall into two camps:
- Fine-tuning (e.g., LoRA): Often struggles to make meaningful updates from just a few samples and can suffer from catastrophic forgetting.
- Retrieval-Augmented Generation (RAG): While effective for simple fact retrieval, it frequently fails when dealing with intricate semantic relationships and complex reasoning required in professional domains.
The Sequential Fusion Framework
To address the limitations of existing schemes, the researchers propose a two-stage Sequential Fusion framework. This method shifts the burden of complex reasoning from the smaller domain-specific model to a high-performing generalist model.
1. Knowledge Extraction
Instead of asking a domain-specific model to parse a complex paper directly, the framework uses a general LLM (such as GPT-4) as a "Relation Extractor." This phase employs a sophisticated prompting strategy consisting of four modules:
- Instruction: Specifies task requirements.
- Reason: Details the reasoning process (Chain-of-Thought).
- Format: Matches output to structured training examples.
- Tips: Provides guidance to avoid common pitfalls and confusion.
2. Knowledge Editing
The extracted structured data (triplets) is passed through a Structured Knowledge Transformation (SKT) module, which converts it into natural language "facts." These facts are then used to update the domain-specific model (such as Llama-2 or Qwen) using In-Context Editing (IKE). Unlike standard fine-tuning, IKE allows the model to integrate new knowledge through specialized prompts without changing the underlying model weights, making the update surgical and reversible.
Experimental Evaluation
The study evaluated the method on two challenging benchmarks:
- DCE (Drug Combo Extraction): A medical dataset involving 1,634 annotated abstracts describing complex n-ary relations between drugs and their combined effects (positive, negative, or undetermined).
- MEE (Moderating Effect-Extraction): A newly introduced dataset containing 100 annotated paragraphs from top economics and management journals, focusing on how one variable moderates the relationship between two others.
Results and Discussion
The Sequential Fusion method outperformed baselines by a wide margin across both domain-specific models tested.
Key performance highlights include:
- Medical (DCE): Achieved 71.7% accuracy, an average gain of 39.1% over traditional methods.
- Economics (MEE): Achieved 75.0% accuracy, an average gain of 45.0%.
- Reliability: While LoRA and RAG struggled with hallucinations and "no answer" responses—with LoRA's "No Answer" rate hitting 70-100% on the MEE dataset—Sequential Fusion provided consistent, effective updates.
Conclusion
The Sequential Fusion approach demonstrates that general-purpose LLMs are better at the reasoning required for extraction than smaller models are at learning directly from raw, messy text. By using the generalist to "clean" the data into structured knowledge, the specialist model can be updated more effectively. Future work will explore integrating even more powerful models for extraction and utilizing Mixture of Experts (MoE) architectures to handle the diversity of domain-specific reasoning required in professional fields.
Summary updated