Abstract: Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization. We attribute this gap to a pre-routing bottleneck: multi-head attention concatenates head-specific signals into a single post-attention router input, forcing routing to act on co-occurring feature compositions rather than separable head channels. We show that this router input simultaneously encodes multiple separately decodable semantic and structural factors with uneven head support, and that different feature compositions induce weakly aligned parameter-gradient directions; as a result, routing maps many distinct compositions to the same route. We quantify this collision effect via a route-wise effective composition number $N_{eff}$ and find that higher $N_{eff}$ is associated with larger old-task loss increases after continual training. Motivated by these findings, we propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions. On TRACE with Qwen3-0.6B/8B, MH-MoE effectively mitigates forgetting, reducing BWT on Qwen3-0.6B from 11.2% (LoRAMoE) to 4.5%.
The Pre-Routing Bottleneck
In continual learning, models often suffer from catastrophic forgetting, where learning new tasks overwrites knowledge from previous ones. While Mixture-of-Experts (MoE) architectures were theoretically expected to solve this by localizing updates to specific "experts," practical implementations in Transformers still struggle.
This research identifies the culprit as the Multi-Head Attention (MHA) mechanism. In standard Transformers, the outputs of multiple attention heads—each specializing in different semantic or structural features—are concatenated into a single vector before being sent to the MoE router. This creates a bottleneck where the router must make a single decision for a complex "cocktail" of features. This leads to composition collisions, where distinct features are forced to share the same experts, causing gradient interference.
Diagnosing Composition Collisions
By using linear probes, the authors demonstrate that different features (such as domain identity, token frequency, or stance) are concentrated in specific attention heads. However, because these signals are merged before routing, the router cannot distinguish between them effectively.
To quantify this, the authors introduce the Effective Composition Number ($N_{eff}$), which measures how many different feature combinations are funneled into the same expert. They find that higher $N_{eff}$ values correlate directly with increased forgetting of old tasks.
MH-MoE: Granular Head-Wise Routing
To address this bottleneck, the authors propose MH-MoE (Multi-Head Mixture-of-Experts). The core idea is to move from token-level routing to head-level routing:
- Head-Aligned Splitting: The post-attention representation is split back into its original head-specific slices.
- Head-Private Routing: Each head is assigned its own independent router and a private bank of experts.
- Increased Granularity: By routing at the head level, the model can isolate updates to specific functional components, drastically reducing the likelihood of unrelated features colliding in the same parameter space.
Why It Works: Gradient Alignment
The effectiveness of MH-MoE is rooted in how it manages gradient updates. The study proves that different feature compositions produce gradients that are "weakly aligned" (nearly orthogonal). When these compositions are forced to share an expert in standard MoE, the update for a new task is likely to damage the parameters supporting an old task.
Experimental Results
MH-MoE was evaluated on the TRACE benchmark, which includes eight diverse tasks ranging from stance detection to code completion, using Qwen3-0.6B and Qwen3-8B models.
Key findings include:
- Significant Forgetting Reduction: On Qwen3-0.6B, the Backward Transfer (BWT) improved from -11.2% (using LoRAMoE) to just -4.5%.
- Improved Performance: Overall Performance (OP) on TRACE increased from 0.378 to 0.467 for the 0.6B model.
- Efficiency: MH-MoE is a practical drop-in replacement, adding only ~1% training latency and ~4% memory overhead.
By increasing routing granularity to match the inherent structure of Multi-Head Attention, MH-MoE provides a robust, task-agnostic architectural solution to catastrophic forgetting in large-scale MoE models.
Summary updated