Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision

Jiaxin Zhang, Zhuohang Li, Kamalika Das, Sricharan KumarNeurIPS 2023 · OpenReview

Introduction

Adapting large language models (LLMs) to specialized domains such as medicine or finance is a critical requirement for enterprise deployment. While general-purpose models like GPT-4 are powerful, smaller, domain-specific models are often preferred for their lower latency, reduced operational costs, and better data privacy. However, the performance of these smaller models depends heavily on high-quality, domain-specific fine-tuning data.

The primary bottleneck in this process is the "data labeling gap." High-fidelity labels from human experts (e.g., physicians or financial analysts) are prohibitively expensive and difficult to scale. Conversely, low-fidelity labels generated by LLMs are inexpensive but prone to hallucinations and lack the technical nuance required for high-stakes applications. This paper introduces Interactive Multi-Fidelity Learning (IMFL), a framework designed to bridge this gap by strategically combining sparse human supervision with automated LLM annotation.

The IMFL Framework

IMFL treats the annotation process as a multi-fidelity learning problem. It uses a small budget of high-fidelity human labels to "teach" an LLM annotator, which then generates a larger volume of "silver" labels for training the target model. The framework is built on three core methodological pillars:

1. Exploration-Exploitation Query (EEQ) Strategy

To maximize the utility of every human label, IMFL moves beyond simple random sampling or basic uncertainty sampling. It employs a two-stage selection process:

  • Exploration: The system uses $k$-means clustering on the unannotated data pool to ensure that the selected samples represent the full breadth of the domain.
  • Exploitation: Within these clusters, the system identifies samples with the highest predictive uncertainty—those the current model is most likely to get wrong. This dual approach ensures the model learns both the general structure of the data and the specific nuances of difficult edge cases.

2. Similarity-Based Prompt Retrieval

One of the most innovative aspects of IMFL is how it uses human labels. Instead of just using them as training data for the final model, human-labeled pairs are stored in a retrieval system. When the LLM is tasked with labeling a new batch of data, the system retrieves the most semantically similar human-labeled examples to serve as in-context demonstrations. This "few-shot" prompting significantly improves the LLM's accuracy by providing it with domain-specific context and expert reasoning patterns.

3. Variable Batch Size (Budget Decay)

Timing is as important as selection. IMFL utilizes a geometric decay for the human annotation budget, front-loading human effort in the early stages of the process. By securing high-quality labels early, the system builds a robust retrieval pool for the LLM annotator, preventing the "cold start" problem where early errors in LLM labeling propagate through the training process.

Experimental Evaluation

The researchers evaluated the IMFL framework across four challenging benchmarks in the financial and medical sectors:

  • Financial News: Financial Phrasebank (FPB) for sentiment analysis and Headline for gold market classification.
  • Biomedical Research: PubMedQA (Yes/No/Maybe) and MedQA (USMLE-style questions).

Key Results

The experiments demonstrated that IMFL provides a massive efficiency boost over traditional labeling methods:

  • Budget Efficiency: Using a total budget of 1,000 labels (200 human + 800 LLM), IMFL consistently outperformed a baseline using 600 human labels (3× the human budget).
  • Performance Parity: On datasets like Headline and PubMedQA, IMFL achieved performance levels nearly identical to a model trained on 1,000 human labels (5× the human budget).
  • LLM Accuracy Gains: The similarity-based prompt retrieval increased the accuracy of the LLM annotator (GPT-3.5) by 6–9% compared to zero-shot or random few-shot baselines.
  • Domain Impact: In the complex MedQA task, the framework showed a +19.95% absolute gain over using LLM labels alone, highlighting the necessity of human-in-the-loop systems for high-stakes domains.

Discussion and Future Work

IMFL demonstrates that human labels have compound value: they serve as both direct training signals and as "teaching tools" for automated annotators. By prioritizing diversity and hardness through the EEQ strategy and stabilizing the process with front-loaded human effort, IMFL offers a viable path for organizations to build high-performing, domain-specific models without the need for massive human workforces.

Future research directions include refining the cost model to account for the varying difficulty of different samples and exploring the use of even more capable models (like GPT-4 or Claude 3) as the low-fidelity source. Additionally, investigating how the framework handles dynamic or evolving data pools could further enhance its utility in real-world, production environments.

Summary updated

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in various tasks. However, their suitability for domain-specific tasks, is limited due to their immense scale at deployment, susceptibility to misinformation, and more importantly, high data annotation costs. We propose a novel Interactive Multi-Fidelity Learning (IMFL) framework for cost-effective development of small domain-specific LMs under limited annotation budgets. Our approach formulates the domain-specific fine-tuning process as a multi-fidelity learning problem, focusing on identifying the optimal acquisition strategy that balances between low-fidelity automatic LLM annotations and high-fidelity human annotations to maximize model performance. We further propose an exploration-exploitation query strategy that enhances annotation diversity and informativeness, incorporating two innovative designs: 1) prompt retrieval that selects in-context examples from human-annotated samples to improve LLM annotation, and 2) variable batch size that controls the order for choosing each fidelity to facilitate knowledge distillation, ultimately enhancing annotation quality. Extensive experiments on financial and medical tasks demonstrate that IMFL achieves superior performance compared with single fidelity annotations. Given a limited budget of human annotation, IMFL significantly outperforms the $\bf 3\times$ human annotation baselines in all four tasks and achieves very close performance as $\bf 5\times$ human annotation on two of the tasks. These promising results suggest that the high human annotation costs in domain-specific tasks can be significantly reduced by employing IMFL, which utilizes fewer human annotations, supplemented with cheaper and faster LLM (e.g., GPT-3.5) annotations to achieve comparable performance.
BibTeX
@inproceedings{zhang2023interactive,
  author = {Jiaxin Zhang and Zhuohang Li and Kamalika Das and Sricharan Kumar},
  title = {{Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=VIaw1XHb4G},
}
Figure