Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers

Anrui Chen, Ruijun Huang, Xin Zhang, Fang Dong, Hengjie Cao, Zhendong Huang, Yifeng Yang, Mengyi Chen, Jixian Zhou, Mingzhi Dong, Yujiang Wang, Jinlong Hou, Qin Lv, Robert P. Dick, Yuan Cheng, Tun Lu, Fan Yang, Li Shang

TL;DR

Mixture-of-Experts (MoE) models are theoretically ideal for continual learning, yet they still suffer from significant catastrophic forgetting. This paper identifies a structural bottleneck in Multi-Head Attention (MHA) that forces distinct features to "collide" during routing, and proposes MH-MoE, a head-wise routing strategy that slashes forgetting by over 50% on the TRACE benchmark.

What Problem Does This Paper Address?

In the quest for Large Language Models (LLMs) that can learn new tasks without forgetting old ones, Mixture-of-Experts (MoE) architectures seem like a silver bullet. By routing different inputs to different "experts," the model should theoretically isolate task-specific updates and prevent interference.

However, the authors observe that even well-balanced MoE models forget substantially. They trace this failure back to the Multi-Head Attention (MHA) bottleneck. In a standard Transformer, the outputs of multiple attention heads—each specializing in different semantic or structural features—are concatenated into a single vector before being sent to the MoE router. This forces the router to make a single decision based on a "mixed" signal, leading to composition collisions: tokens with vastly different needs are sent to the same expert, causing gradient interference and, ultimately, catastrophic forgetting.

Methods

The authors first perform a diagnostic analysis using linear probes to prove that the router input is "multiplexed"—it simultaneously carries distinct signals like domain identity, stance, and token frequency. They then introduce two key innovations:

  1. $N_{eff}$ (Effective Composition Number): A new metric to quantify how many distinct feature compositions are being crammed into a single route. They show a direct correlation: the higher the $N_{eff}$, the worse the forgetting.
  2. MH-MoE (Multi-Head Mixture-of-Experts): Instead of one routing decision for the entire concatenated vector, MH-MoE splits the post-attention representation back into its original head-aligned slices. Each slice is routed independently to its own "private" expert bank. This increases routing granularity and ensures that features originating in different heads don't interfere with each other during the expert update process.

Datasets

The researchers evaluated their approach using the TRACE benchmark, a suite of eight diverse continual learning tasks:

  • C-STANCE (Chinese stance detection)
  • FOMC (Federal Open Market Committee communications)
  • MeetingBank (Summarization)
  • Py150 (Code completion)
  • ScienceQA (Question answering)
  • NumGLUE-cm & NumGLUE-ds (Arithmetic reasoning)
  • 20Minuten (German text simplification)

Experiments were conducted on Qwen3-0.6B and Qwen3-8B backbones.

Key Insights and Ideas

  • MHA is a Feature Blender: The study confirms that different attention heads specialize in different features (e.g., one head might track position while another tracks sentiment). Concatenating them before routing creates a "jumbled" signal that the router cannot effectively separate.

    Figure 1. The router input multiplexes multiple decodable features.
  • Gradient Directions Diverge: The authors found that different feature compositions induce parameter-gradient directions with very low cosine similarity. When these compositions collide in the same expert, the resulting update is a "compromise" that helps neither task well and actively harms old ones.

    Figure 3. Different feature compositions induce distinct gradient directions.
  • Granularity is the Cure: By moving from a single global routing decision to $H$ head-wise decisions, MH-MoE effectively increases the "resolution" of the model's conditional computation, allowing for much cleaner parameter isolation.

Key Results

MH-MoE consistently outperformed standard MoE baselines (like LoRAMoE) and traditional continual learning methods (like EWC or GEM):

  • Reduced Forgetting: On Qwen3-0.6B, MH-MoE reduced Backward Transfer (BWT)—a measure of forgetting—from -11.2% to -4.5%.
  • Better Overall Performance: The Overall Performance (OP) score jumped from 0.378 to 0.467 on the 0.6B model.
  • Efficiency: Despite the increased routing complexity, MH-MoE only adds about 1% training latency and a negligible 4% increase in peak memory usage compared to standard LoRAMoE.
  • Robustness: The benefits persisted across different task orderings, proving that the mitigation of forgetting isn't just a fluke of a specific curriculum.
Figure 4. Composition mixing persists across layers. Old- task mass-weighted average effective composition number Neff (Eq. (9)) across routes in each MoE layer. Values substantially above 1 indicate that routes typically aggregate multiple feature compositions under the old-task distribution.

Limitations and Future Work

  • Private vs. Shared Experts: MH-MoE currently uses private expert banks for each head. The authors suggest that exploring shared expert banks with head-wise routing might offer different trade-offs in parameter efficiency.
  • Task-Agnosticism: While MH-MoE is naturally task-agnostic (it doesn't need task IDs), the study primarily focuses on sequential task learning. Its performance in purely "blended" multi-task streams remains an area for further exploration.
  • Extreme Scaling: While tested up to 8B parameters, the behavior of composition collisions in "frontier" models (70B+) is still an open question.

Summary updated

Abstract

Mixture-of-Experts (MoE) architectures are often considered a natural fit for continual learning because sparse routing should localize updates and reduce interference, yet MoE Transformers still forget substantially even with sparse, well-balanced expert utilization. We attribute this gap to a pre-routing bottleneck: multi-head attention concatenates head-specific signals into a single post-attention router input, forcing routing to act on co-occurring feature compositions rather than separable head channels. We show that this router input simultaneously encodes multiple separately decodable semantic and structural factors with uneven head support, and that different feature compositions induce weakly aligned parameter-gradient directions; as a result, routing maps many distinct compositions to the same route. We quantify this collision effect via a route-wise effective composition number $N_{eff}$ and find that higher $N_{eff}$ is associated with larger old-task loss increases after continual training. Motivated by these findings, we propose MH-MoE, which performs head-wise routing over sub-representations to increase routing granularity and reduce composition collisions. On TRACE with Qwen3-0.6B/8B, MH-MoE effectively mitigates forgetting, reducing BWT on Qwen3-0.6B from 11.2% (LoRAMoE) to 4.5%.
BibTeX
@inproceedings{chen2026multi,
  author = {Anrui Chen and Ruijun Huang and Xin Zhang and Fang Dong and Hengjie Cao and Zhendong Huang and Yifeng Yang and Mengyi Chen and Jixian Zhou and Mingzhi Dong and Yujiang Wang and Jinlong Hou and Qin Lv and Robert P. Dick and Yuan Cheng and Tun Lu and Fan Yang and Li Shang},
  title = {{Multi-Head Attention as a Source of Catastrophic Forgetting in MoE Transformers}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=iXhFXwIbvi},
}
Figure