Abstract
Training Large Language Models (LLMs) remains a resource-intensive challenge. While Mixture of Experts (MoE) architectures improve inference efficiency, their training phase is often a bottleneck. This paper presents MoFE (Mixture of Frozen Experts), an architecture that integrates MoE with Parameter-Efficient Fine-Tuning (PEFT) principles. By freezing the Feed-Forward Network (FFN) layers of the experts and training only the base architecture and router, MoFE achieves superior performance and faster training speeds compared to established PEFT methods like LoRA.
Introduction
The current landscape of LLM development is defined by a "scaling vs. efficiency" dilemma. Scaling laws suggest that larger models perform better, yet the cost of training and deploying them is prohibitive. MoE architectures, such as Mixtral, address inference costs by activating only a subset of parameters per token. However, training these experts from scratch remains expensive. Conversely, PEFT methods like LoRA reduce trainable parameters but often fail to match full fine-tuning performance or effectively incorporate deep domain expertise. MoFE bridges this gap by utilizing pre-existing, frozen experts within an MoE framework.
Architecture and Methodology
The MoFE architecture consists of three primary components: a base model, a set of frozen experts, and a trainable router.
- Base Model: Provides the foundational trainable parameters, including embeddings and self-attention layers. In this study, TinyLlama (1.1B) served as the base.
- Frozen Experts: These are FFN blocks sourced from other models. Their weights are frozen during training, ensuring that the number of trainable parameters (approximately 0.34B) remains constant regardless of the number of experts.
- Router (Gate): A trainable linear layer that selects the top m experts (typically m=2) to activate for each token.
Experimental Results
The researchers evaluated MoFE across general knowledge (MMLU), medical (MedMCQA, PubMedQA), and finance domains.
Performance vs. PEFT Methods
A key finding is that MoFE consistently outperforms standard PEFT methods such as LoRA, QLoRA, and DoRA across various model scales. Despite having more trainable parameters than LoRA (340M vs. ~2.4M), MoFE demonstrates superior training efficiency.
As shown in the results, MoFE achieved a training time of just 6 hours for the "Small" configuration, whereas LoRA required 13 hours. This efficiency gain is attributed to MoFE updating existing layers directly rather than computing additional adapter paths.
Domain Expertise and Scaling
The study demonstrated that MoFE effectively facilitates "plug-and-play" expertise. Integrating FFNs from models pre-trained on medical data immediately boosted performance in medical benchmarks. For instance, increasing the number of specialized medical experts from zero to four improved MedMCQA scores from 0.3488 to 0.3636.
Discussion and Insights
The research yielded several counter-intuitive insights:
- Post-Pretraining Risks: Attempting to "post-pretrain" (training on raw text before instruction tuning) with frozen experts led to performance degradation. The static nature of the FFNs prevents them from adapting to new knowledge, causing misalignment with the trainable attention layers.
- Base Model Neutrality: General-purpose base models outperformed domain-specific ones in multi-domain tasks, suggesting that the base model should remain a generalist while experts handle specialized knowledge.
Conclusion
MoFE offers a promising path for scaling LLMs efficiently. By leveraging frozen experts, it provides a high-performance alternative to traditional fine-tuning and PEFT methods, significantly reducing training time while maintaining the ability to integrate specialized domain knowledge. Future work will explore the scalability of MoFE with larger foundational models and a broader array of expert domains.
Citation: Seo, J., Kim, J., & Shin, H. (2025). MoFE: Mixture of Frozen Experts Architecture. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics (Industry Track), 340–348.
Summary updated