Abstract
Considering deep neural networks as manifold mappers, the pretrain-then-fine-tune paradigm can be interpreted as a two-stage process: pretrain establishes a broad knowledge base, and fine-tune adjusts the model parameters to activate specific neural pathways to align with the target manifold. Although prior fine-tuning approaches demonstrate success, their rigid parameter space limits their ability to dynamically activate appropriate neural pathways, rendering them ill-equipped to adapt flexibly to the diverse and evolving data distributions. In light of this view, we propose a novel approach, Mixture of Expert Prompt Tuning (MEPT), as an effective and efficient manifold-mapping framework. MEPT leverages the Mixture of Experts architecture by integrating multiple prompt experts to adaptively learn diverse and non-stationary data distributions. Empirical evaluations demonstrate that MEPT outperforms several state-of-the-art parameter efficient baselines on SuperGLUE, achieving notable improvements in mean accuracy (e.g., 1.94%) while significantly reducing activated prompts by 79.25%. The effectiveness of MEPT is further supported by theoretical insights from manifold learning and validated through neural activation pathway visualization results. Our code is avaliable at https://runjia.tech/emnlp_mept/.
Introduction
The "pretrain-then-fine-tune" paradigm has become the standard for leveraging large language models (LLMs). However, as models scale to billions of parameters, full fine-tuning becomes computationally prohibitive. This has led to the development of Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and Prompt Tuning.
Despite their efficiency, existing PEFT methods often employ a fixed parameter space—such as a single set of prompts—for all inputs. This rigidity limits the model's capacity to handle diverse data distributions that reside on complex, overlapping manifolds. We argue that a static prompt lacks the representational flexibility to navigate these distinct structures effectively. To address this, we introduce Mixture of Expert Prompt Tuning (MEPT), which reimagines prompt tuning through the lens of manifold mapping.
The MEPT Framework
MEPT treats fine-tuning as a dynamic manifold-mapping problem. Instead of a single static prompt, MEPT utilizes a sparse Mixture of Experts (MoE) architecture integrated into the transformer layers.
The framework consists of several key innovations:
- Router Experts (Specialists): Multiple prompt embeddings are available at each layer. A small linear "router" network selects the most appropriate expert (Top-1 activation) for each input. This allows the model to activate specialized neural pathways based on the task type (e.g., question answering vs. natural language inference).
- Shared Experts (Knowledge Consolidators): To prevent redundancy, a shared expert remains active for all inputs. This consolidates general linguistic patterns, allowing the router experts to focus on task-specific nuances.
- Sparse Activation: By activating only one router expert per layer, MEPT maintains low computational overhead during inference while benefiting from a significantly larger pool of total parameters.
Experimental Results and Manifold Analysis
We evaluated MEPT on the SuperGLUE benchmark using T5-Base (220M), T5-Large (770M), and Llama 3.2 (1B) models.
Key Findings:
- Superior Performance: MEPT consistently outperformed vanilla prompt tuning and other PEFT baselines. On T5-Base, it even surpassed full fine-tuning by 0.18%. On Llama 3.2 (1B), it achieved a 1.94% improvement in mean accuracy over the best existing PEFT method.
- Efficiency: Compared to P-Tuning v2, MEPT reduced the number of activated prompts by 79.25% while delivering higher accuracy.
- Manifold Separability: Theoretical insights from manifold learning suggest that MEPT helps the model "collapse" task-relevant information into separable clusters. Visualizations (e.g., t-SNE) confirm that MEPT creates cleaner, more compact clusters in the final layers compared to static methods.
Conclusion
MEPT offers a dynamic and efficient alternative to traditional prompt tuning. By viewing LLMs as manifold mappers and employing a sparse MoE architecture, MEPT achieves state-of-the-art performance with a fraction of the activated parameters. This approach not only improves accuracy but also provides a more flexible framework for adapting to the diverse data distributions encountered in real-world NLU tasks.
Summary updated