Abstract
Continual instruction tuning (CIT) requires multimodal large language models (MLLMs) to adapt to a stream of tasks without forgetting prior capabilities. A common strategy is to isolate updates by routing inputs to different LoRA experts. However, existing LoRA-based Mixture-of-Experts (MoE) methods often jointly update the router and experts in an indiscriminate way, causing the router's preferences to co-drift with experts' adaptation pathways and gradually deviate from early-stage input-expert specialization. We term this phenomenon Misaligned Co-drift, which blurs expert responsibilities and exacerbates forgetting. To address this, we introduce the pathway activation subspace (PASs), a LoRA-induced subspace that reflects which low-rank pathway directions an input activates in each expert, providing a capability-aligned coordinate system for routing and preservation. Based on PASs, we propose a fixed-capacity PASs-based MoE-LoRA method with two components: PAS-guided Reweighting, which calibrates routing using each expert's pathway activation signals, and PAS-aware Rank Stabilization, which selectively stabilizes rank directions important to previous tasks. Experiments on a CIT benchmark show that our approach consistently outperforms a range of conventional continual learning baselines and MoE-LoRA variants in both accuracy and anti-forgetting without adding parameters.
Introduction: The Challenge of Misaligned Co-drift
In the realm of Continual Instruction Tuning (CIT), Multimodal Large Language Models (MLLMs) must learn a sequence of tasks without suffering from catastrophic forgetting. A popular architectural choice is the Mixture-of-Experts (MoE) using Low-Rank Adaptation (LoRA), where a router directs inputs to specific experts. However, standard MoE-LoRA implementations suffer from a phenomenon the authors identify as Misaligned Co-drift.
As the model is tuned on a new task $n+1$, both the router and the experts undergo updates. This simultaneous change causes the router's selection logic to drift away from the experts' actual capabilities established in task $n$. This "double-sided drift" blurs the boundaries of expert specialization, making it difficult for the model to maintain performance on previous tasks.
Proposed Method: PASs-MoE
To mitigate this, the authors introduce Pathway Activation Subspaces (PASs). PASs leverage the internal structure of LoRA experts—specifically the down-projection matrix $A$—to create a coordinate system that reflects which low-rank pathways an input activates.
The PASs-MoE framework consists of two primary components:
- PAS-guided Reweighting (PASs-RW): Instead of relying on an independent neural network for routing, PASs-RW uses "implicit self-routing." It calculates the "activation energy" of an input within each expert's internal pathways. If an expert's circuitry reacts strongly to an input, it receives a higher routing weight. This ensures that routing is physically grounded in the expert's current capabilities, eliminating the possibility of co-drift.
- PAS-aware Rank Stabilization (PASs-RS): Recognizing that task-specific knowledge is often sparse within an expert's rank directions, PASs-RS identifies and stabilizes the specific "wires" (rows of the LoRA matrix) that were critical for past tasks. By applying a selective penalty, the model protects historically important directions while allowing others to adapt to new tasks.
Experimental Evaluation
The researchers evaluated PASs-MoE on the MLLM-CTBENCH, a benchmark featuring seven diverse tasks (Math QA, Arts VQA, Economics QA, Science VQA, Medicine VQA, and OCR VQA) using LLaVA-v1.5-7B as the backbone.
Key Results
- Performance Gains: PASs-MoE achieved an Average Performance (AP) of 48.46%, a significant improvement over the best MoE-LoRA baseline (43.36%).
- Superior Anti-Forgetting: The method demonstrated remarkable stability. In the Arts VQA task, while baselines showed heavy forgetting (Backward Transfer of -11.45), PASs-MoE achieved positive transfer (+10.43).
- Parameter Efficiency: Notably, PASs-MoE maintains a fixed capacity. It achieves these state-of-the-art results without increasing the parameter count or inference budget as the number of tasks grows.
Conclusion
PASs-MoE offers a principled solution to the stability-plasticity dilemma in continual learning by grounding routing logic in the model's internal activation pathways. The authors' approach demonstrates that by aligning routing with internal expert capabilities, MLLMs can learn indefinitely without losing their previously acquired knowledge.
Summary updated