TL;DR
This paper introduces SliceMoE, a novel approach to Mixture-of-Experts (MoE) models that routes contiguous slices of a token's embedding vector to experts, rather than the entire token. This fine-grained routing leads to improved load balancing, better parameter utilization, and more interpretable expert specialization, while achieving faster inference and lower perplexity compared to existing MoE methods.
What Problem Does This Paper Address?
Transformer models, particularly large ones, are often scaled using Mixture-of-Experts (MoE) layers. These layers activate only a sparse subset of "expert" feed-forward networks (FFNs) for each token, making them computationally efficient. However, traditional token-level MoE suffers from several issues:
- Capacity Bottlenecks & Load Imbalance: Popular experts can become overloaded, while others remain underutilized, wasting parameters and leading to uneven computation.
- Limited Specialization: Assigning an entire token's semantic spectrum to a single expert restricts its ability to develop deep, narrow expertise.
- Latency Spikes: Uneven expert utilization can lead to unpredictable inference times.
The authors hypothesize that different parts (slices) of a token's embedding vector capture distinct, partially independent information (e.g., syntactic vs. semantic cues). They aim to leverage this sub-token diversity for more efficient and effective conditional computation.
Methods
SliceMoE's core innovation is slice-level routing. Instead of routing an entire token's embedding vector, the authors partition each token's $d$-dimensional embedding into $S$ contiguous slices. For each slice, a lightweight, shared router (a small Multi-Layer Perceptron or MLP) predicts the top-$k$ experts. Each expert then processes only its assigned slice. The outputs from these slices are re-assembled to form the final token representation.
Key methodological contributions include:
- Slice-Level Routing: The fundamental unit of routing is shifted from a token to a slice of its embedding.
- Slice-Level Capacity Loss: A novel loss function is introduced to encourage balanced load distribution across experts at the slice level, defined as the squared coefficient of variation of slice assignments per expert. This is shown to be more effective than token-level capacity losses.
- Cross-Slice Dropout: To prevent over-reliance on specific slice-expert pairings during training, a fraction of the top-k assignment probabilities for each slice are randomly zeroed out, forcing the router to explore alternatives.
- Fused Kernels for Efficiency: To maintain GPU efficiency, slices destined for the same expert are dynamically grouped and stacked, allowing expert FFNs to be processed using efficient batched matrix multiply operations, comparable to dense layers.
Datasets
The paper evaluates SliceMoE on a variety of tasks and datasets:
- Language Modelling: WikiText-103 (WT-103)
- Machine Translation: WMT-21 English–German
- Text Classification: AG NEWS, DBPEDIA-14, and EMOTION (from HuggingFace Datasets)
- Synthetic Dataset: A custom 64-dimensional dataset was used for initial toy experiments to demonstrate the model's behavior on distinct feature segments.
Key Insights and Ideas
- Sub-token structure matters: The paper strongly suggests that embedding vectors are not monolithic but contain structured information that can be exploited by routing different parts to different experts.
- Fine-grained routing improves balance: Shifting from token-level to slice-level routing significantly smooths expert load distribution, leading to higher Expert Load Entropy (ELE), a measure of balance.
- Interpretable specialization: SliceMoE experts show more granular specialization. For instance, one expert might focus on financial terms, another on sports, providing a direct path for debugging and understanding model behavior.
- Optimal slice granularity: There's an optimal number of slices ($S=8$ for a 768-dim embedding was found to be effective), where too few limit fine-grained routing and too many can increase overhead or fragment information.
- Contiguous slices are better: Routing contiguous blocks of the embedding vector performs better than routing randomly shuffled slices, supporting the hypothesis that local, coherent information is captured within these blocks.
Key Results
- Performance: SliceMoE achieves up to 1.7x faster inference than dense baselines and 12-18% lower perplexity than parameter-matched token-MoE models on language modeling and classification tasks.
- Load Balancing: SliceMoE consistently achieves near-optimal expert load balance (ELE ≈ 0.95-0.97), outperforming token-MoE which often shows a trade-off between accuracy and balance.
- Accuracy: On AG NEWS and DBPEDIA-14, SliceMoE consistently outperforms TokenMoE by 2-4 percentage points in accuracy.
- Efficiency: Achieves performance comparable to dense layers on capable hardware (A100 GPUs) due to fused kernels.
- Robustness: The routing mechanism is robust to noise, with accuracy remaining stable until noise standard deviation exceeds 0.5.
Limitations and Future Work
- Scalability of Router: The router MLP's input dimension scales with $d/S$. For very large $S$ or many experts $E$, routing computation could become a bottleneck. Future work could explore hierarchical routing or dynamic slice counts.
- Hyperparameter Sensitivity: The performance is sensitive to hyperparameters like the number of slices ($S$), top-$k$ experts, capacity loss weight ($\alpha$), and cross-slice dropout rate. Optimal values may depend on embedding dimension and task.
- Hardware Dependency: The efficiency gains from fused kernels are most pronounced on modern GPUs (e.g., A100s). Broader hardware compatibility and optimized open-source kernels are needed for wider adoption.
- Classification Setup: Classification experiments used a frozen DistilBERT encoder. Full end-to-end fine-tuning of large models with SliceMoE is an important next step.
- Interpretability Metrics: Current interpretability analysis relies on specific metrics and qualitative examples. Developing more comprehensive quantitative metrics for expert specialization is an open research area.
- Future Directions: Exploring hierarchical routing, adaptive slice counts, and porting fused kernels to more accelerators are suggested.
Summary updated