Introduction
The rapid expansion of the open-source community has produced a vast repository of specialized Low-Rank Adaptation (LoRA) adapters and instruction datasets. However, for researchers targeting niche tasks with limited data ($K$-shot), the abundance of resources presents a significant selection challenge. Identifying the most effective models and data samples from this "open knowledge" goldmine is non-trivial. This paper introduces Rocket, an efficient pipeline designed to navigate this landscape using minimal human-verified data to construct high-performing Mixture-of-Experts (MoE) systems.
Methodology: The Rocket Pipeline
Rocket addresses the limitations of general-purpose LLMs in vertical domains—such as specialized medicine or complex law—by intelligently repurposing existing open-source assets. The framework follows a structured four-step process:
- LoRA Bank Construction: A library of specialized LoRA adapters (e.g., LLaMA-2, Mistral) is assembled, covering diverse tasks from mathematical reasoning to tool usage.
- K-shot Guided Expert Selection: Rather than relying solely on raw accuracy, Rocket introduces Reasoning Perplexity. By using a teacher model to generate Chain-of-Thought (CoT) rationales for $K$-shot samples, the system measures how well candidate models align with expert reasoning. Furthermore, Group Diversity is calculated via parameter-level cosine similarity to ensure the selected experts offer complementary capabilities.
- MoE Initialization: The most diverse, low-perplexity models are integrated into a Mixture-of-Experts system featuring a learnable gating network.
- Sim-Div Data Selection: To augment the training set without overfitting, a "Similarity-First, Diversity-Aware" strategy retrieves open-source instructions semantically related to the $K$-shot data while maintaining a broad distribution to prevent signal dilution.
Key Insights: Reasoning and Diversity
A core contribution of this work is the shift from outcome-based evaluation to process-based evaluation. The researchers found that a model's perplexity on a step-by-step rationale is a far more robust indicator of true expertise than simple multiple-choice accuracy.
Additionally, the "Diversity Mandate" ensures that the MoE system avoids redundancy. By selecting models with diverse parameter weights, the resulting system covers a wider knowledge boundary, allowing for token-wise cooperation where different experts handle logic versus technical terminology dynamically.
Experimental Results
The Rocket framework was evaluated against six challenging benchmarks, including ARC-Challenge, PiQA, MBPP (coding), and GSM8K (math).
- Performance Gains: Using a Mistral-7B backbone, Rocket achieved an average accuracy of 72.77%, a substantial increase over the base model's 54.90%.
- Competitive Edge: Rocket outperformed state-of-the-art methods such as PHATGOOSE (65.36%) and Arrow (71.53%).
- Data Efficiency: The system maintains high performance with as few as 5 human-verified samples ($K=5$), demonstrating its viability for rapid domain-specific deployment.
Conclusion
Rocket demonstrates that the "open knowledge" within the LLM ecosystem can be systematically harvested to build task experts. While currently focused on LoRA adapters, future work may extend this to full-parameter models and integrate human-readable metadata to further refine the selection process. By prioritizing reasoning depth and structural diversity, Rocket provides a scalable path toward specialized artificial intelligence.
Summary updated