Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models

Yuncheng Yang, Yulei Qin, Tong Wu, Zihan Xu, Gang Li, Pengcheng Guo, Hang Shao, Yucheng Shi, Ke Li, Xing Sun, Jie Yang, Yun GuarXiv 2024 · DOI · OpenReview

Introduction

The rapid expansion of the open-source community has produced a vast repository of specialized Low-Rank Adaptation (LoRA) adapters and instruction datasets. However, for researchers targeting niche tasks with limited data ($K$-shot), the abundance of resources presents a significant selection challenge. Identifying the most effective models and data samples from this "open knowledge" goldmine is non-trivial. This paper introduces Rocket, an efficient pipeline designed to navigate this landscape using minimal human-verified data to construct high-performing Mixture-of-Experts (MoE) systems.

Methodology: The Rocket Pipeline

Rocket addresses the limitations of general-purpose LLMs in vertical domains—such as specialized medicine or complex law—by intelligently repurposing existing open-source assets. The framework follows a structured four-step process:

  1. LoRA Bank Construction: A library of specialized LoRA adapters (e.g., LLaMA-2, Mistral) is assembled, covering diverse tasks from mathematical reasoning to tool usage.
  2. K-shot Guided Expert Selection: Rather than relying solely on raw accuracy, Rocket introduces Reasoning Perplexity. By using a teacher model to generate Chain-of-Thought (CoT) rationales for $K$-shot samples, the system measures how well candidate models align with expert reasoning. Furthermore, Group Diversity is calculated via parameter-level cosine similarity to ensure the selected experts offer complementary capabilities.
  3. MoE Initialization: The most diverse, low-perplexity models are integrated into a Mixture-of-Experts system featuring a learnable gating network.
  4. Sim-Div Data Selection: To augment the training set without overfitting, a "Similarity-First, Diversity-Aware" strategy retrieves open-source instructions semantically related to the $K$-shot data while maintaining a broad distribution to prevent signal dilution.
The Rocket pipeline for constructing task experts using K-shot guided model and data selection.

Key Insights: Reasoning and Diversity

A core contribution of this work is the shift from outcome-based evaluation to process-based evaluation. The researchers found that a model's perplexity on a step-by-step rationale is a far more robust indicator of true expertise than simple multiple-choice accuracy.

Correlation between reasoning perplexity and performance, highlighting the effectiveness of the Mixture-of-Expert SFT.

Additionally, the "Diversity Mandate" ensures that the MoE system avoids redundancy. By selecting models with diverse parameter weights, the resulting system covers a wider knowledge boundary, allowing for token-wise cooperation where different experts handle logic versus technical terminology dynamically.

Experimental Results

The Rocket framework was evaluated against six challenging benchmarks, including ARC-Challenge, PiQA, MBPP (coding), and GSM8K (math).

  • Performance Gains: Using a Mistral-7B backbone, Rocket achieved an average accuracy of 72.77%, a substantial increase over the base model's 54.90%.
  • Competitive Edge: Rocket outperformed state-of-the-art methods such as PHATGOOSE (65.36%) and Arrow (71.53%).
  • Data Efficiency: The system maintains high performance with as few as 5 human-verified samples ($K=5$), demonstrating its viability for rapid domain-specific deployment.

Conclusion

Rocket demonstrates that the "open knowledge" within the LLM ecosystem can be systematically harvested to build task experts. While currently focused on LoRA adapters, future work may extend this to full-parameter models and integrate human-readable metadata to further refine the selection process. By prioritizing reasoning depth and structural diversity, Rocket provides a scalable path toward specialized artificial intelligence.

Summary updated

Abstract

The cultivation of expertise for large language models (LLMs) to solve tasks of specific areas often requires special-purpose tuning with calibrated behaviors on the expected stable outputs. To avoid huge cost brought by manual preparation of instruction datasets and training resources up to hundreds of hours, the exploitation of open knowledge including a wealth of low rank adaptation (LoRA) models and instruction datasets serves as a good starting point. However, existing methods on model and data selection focus on the performance of general-purpose capabilities while neglecting the knowledge gap exposed in domain-specific deployment. In the present study, we propose to bridge such gap by introducing few human-annotated samples (i.e., K-shot) for advancing task expertise of LLMs with open knowledge. Specifically, we develop an efficient and scalable pipeline to cost-efficiently produce task experts where K-shot data intervene in selecting the most promising expert candidates and the task-relevant instructions. A mixture-of-expert (MoE) system is built to make the best use of individual-yet-complementary knowledge between multiple experts. We unveil the two keys to the success of a MoE system, 1) the abidance by K-shot, and 2) the insistence on diversity. For the former, we ensure that models that truly possess problem-solving abilities on K-shot are selected rather than those blind guessers. Besides, during data selection, instructions that share task-relevant contexts with K-shot are prioritized. For the latter, we highlight the diversity of constituting experts and that of the fine-tuning instructions throughout the model and data selection process. Extensive experimental results confirm the superiority of our approach over existing methods on utilization of open knowledge across various tasks. Our codes will be available at https://github.com/Yaphabates/Rocket.
BibTeX
@misc{yang2024leveraging,
  author = {Yuncheng Yang and Yulei Qin and Tong Wu and Zihan Xu and Gang Li and Pengcheng Guo and Hang Shao and Yucheng Shi and Ke Li and Xing Sun and Jie Yang and Yun Gu},
  title = {{Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models}},
  year = {2024},
  eprint = {2408.15915},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.15915},
}
Figure