TL;DR
Mixture-of-Experts (MoE) models are theoretically ideal for continual learning, yet they still suffer from significant catastrophic forgetting. This paper identifies a structural bottleneck in Multi-Head Attention (MHA) that forces distinct features to "collide" during routing, and proposes MH-MoE, a head-wise routing strategy that slashes forgetting by over 50% on the TRACE benchmark.
What Problem Does This Paper Address?
In the quest for Large Language Models (LLMs) that can learn new tasks without forgetting old ones, Mixture-of-Experts (MoE) architectures seem like a silver bullet. By routing different inputs to different "experts," the model should theoretically isolate task-specific updates and prevent interference.
However, the authors observe that even well-balanced MoE models forget substantially. They trace this failure back to the Multi-Head Attention (MHA) bottleneck. In a standard Transformer, the outputs of multiple attention heads—each specializing in different semantic or structural features—are concatenated into a single vector before being sent to the MoE router. This forces the router to make a single decision based on a "mixed" signal, leading to composition collisions: tokens with vastly different needs are sent to the same expert, causing gradient interference and, ultimately, catastrophic forgetting.
Methods
The authors first perform a diagnostic analysis using linear probes to prove that the router input is "multiplexed"—it simultaneously carries distinct signals like domain identity, stance, and token frequency. They then introduce two key innovations:
- $N_{eff}$ (Effective Composition Number): A new metric to quantify how many distinct feature compositions are being crammed into a single route. They show a direct correlation: the higher the $N_{eff}$, the worse the forgetting.
- MH-MoE (Multi-Head Mixture-of-Experts): Instead of one routing decision for the entire concatenated vector, MH-MoE splits the post-attention representation back into its original head-aligned slices. Each slice is routed independently to its own "private" expert bank. This increases routing granularity and ensures that features originating in different heads don't interfere with each other during the expert update process.
Datasets
The researchers evaluated their approach using the TRACE benchmark, a suite of eight diverse continual learning tasks:
- C-STANCE (Chinese stance detection)
- FOMC (Federal Open Market Committee communications)
- MeetingBank (Summarization)
- Py150 (Code completion)
- ScienceQA (Question answering)
- NumGLUE-cm & NumGLUE-ds (Arithmetic reasoning)
- 20Minuten (German text simplification)
Experiments were conducted on Qwen3-0.6B and Qwen3-8B backbones.
Key Insights and Ideas
-
MHA is a Feature Blender: The study confirms that different attention heads specialize in different features (e.g., one head might track position while another tracks sentiment). Concatenating them before routing creates a "jumbled" signal that the router cannot effectively separate.
Figure 1. The router input multiplexes multiple decodable features. -
Gradient Directions Diverge: The authors found that different feature compositions induce parameter-gradient directions with very low cosine similarity. When these compositions collide in the same expert, the resulting update is a "compromise" that helps neither task well and actively harms old ones.
Figure 3. Different feature compositions induce distinct gradient directions. -
Granularity is the Cure: By moving from a single global routing decision to $H$ head-wise decisions, MH-MoE effectively increases the "resolution" of the model's conditional computation, allowing for much cleaner parameter isolation.
Key Results
MH-MoE consistently outperformed standard MoE baselines (like LoRAMoE) and traditional continual learning methods (like EWC or GEM):
- Reduced Forgetting: On Qwen3-0.6B, MH-MoE reduced Backward Transfer (BWT)—a measure of forgetting—from -11.2% to -4.5%.
- Better Overall Performance: The Overall Performance (OP) score jumped from 0.378 to 0.467 on the 0.6B model.
- Efficiency: Despite the increased routing complexity, MH-MoE only adds about 1% training latency and a negligible 4% increase in peak memory usage compared to standard LoRAMoE.
- Robustness: The benefits persisted across different task orderings, proving that the mitigation of forgetting isn't just a fluke of a specific curriculum.
Limitations and Future Work
- Private vs. Shared Experts: MH-MoE currently uses private expert banks for each head. The authors suggest that exploring shared expert banks with head-wise routing might offer different trade-offs in parameter efficiency.
- Task-Agnosticism: While MH-MoE is naturally task-agnostic (it doesn't need task IDs), the study primarily focuses on sequential task learning. Its performance in purely "blended" multi-task streams remains an area for further exploration.
- Extreme Scaling: While tested up to 8B parameters, the behavior of composition collisions in "frontier" models (70B+) is still an open question.
Summary updated