Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing

Meng Lou, Stanley Yu, Yizhou Yu

Fine-tuning massive vision models for specific downstream tasks presents a significant computational dilemma. While "full fine-tuning"—updating every single parameter—yields the highest accuracy, it is computationally exorbitant and requires storing a complete model copy for every task. Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA or AdaptFormer, offer a pragmatic middle ground by training only a minuscule fraction of parameters. However, these conventional PEFT methods frequently falter on "dense" prediction tasks like object detection or semantic segmentation, where the model must decipher intricate spatial relationships.

A recent paper, "Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing," introduces ParaX, a novel methodology that bridges this performance gap by fundamentally rethinking how adapter weights are generated and applied.

The Challenge: Static Weights and Redundant Features

Existing PEFT methods face two primary hurdles when applied to dense prediction tasks:

  1. Representation Deficiency: Most traditional adapters utilize "static" weights. Once trained, they apply the exact same mathematical transformation regardless of whether the input image is a simple, centered object or a highly complex street scene. This lack of input-dependency severely limits the model's expressive capacity.
  2. Feature Redundancy: Adapters are typically inserted layer-by-layer in complete isolation. Consequently, different layers often learn nearly identical, redundant features, which is highly inefficient for complex spatial reasoning.

For tasks like semantic segmentation, a model requires "customized" weights capable of adapting to the specific geometry and context of an individual input image.

The ParaX Solution: Dynamic Parameter Routing

To overcome these limitations, ParaX introduces a novel Mixture-of-Experts (MoE) architecture. However, instead of routing data to different expert sub-networks as in traditional MoE, ParaX routes parameters to the model.

  • Shared Expert Center: Rather than assigning a private adapter to each layer, ParaX establishes a central pool of trainable parameter matrices (experts) that are shared across an entire stage of the network.
  • Dynamic Parameter Routing: During every forward pass, a lightweight "router" network analyzes the input features and generates a set of gating coefficients. These coefficients linearly combine the experts into a single, dynamic weight matrix tailored specifically for that exact input and layer.
  • Multi-scale Spatial Mixing: To effectively process objects of varying sizes, ParaX incorporates dynamic depthwise convolutions with diverse kernel sizes (e.g., 3x3, 5x5, 7x7). These are also dynamically generated from the shared expert center.
  • Implicit Cross-layer Interaction: Because the experts are shared, gradients from multiple layers are aggregated to update the same set of parameters during backpropagation. This mechanism forces the layers to coordinate, diversifying their feature representations.

Key Insights

The architecture of ParaX yields several fascinating theoretical and practical insights:

  • Input-Dependency is Crucial: Static adapters are too rigid for dense vision. By making weights a function of the input, ParaX allows the model to dynamically focus its parameters on the specific textures or shapes present in a given image.
  • Sharing Reduces Redundancy: Counter-intuitively, sharing parameters across layers actually increases feature diversity. Using Centered Kernel Alignment (CKA), the authors demonstrate that ParaX layers produce less redundant information than standard adapters, approaching the rich diversity seen in full fine-tuning.
  • Holistic Gradient Aggregation: ParaX creates a direct pathway for gradients to flow across layers via the shared experts, bypassing the sequential bottleneck of standard backpropagation and allowing for more holistic model updates.

Experimental Validation and Results

The authors evaluated ParaX across a rigorous suite of benchmarks, including dense prediction (ADE20K, COCO2017), general classification (CIFAR-100, SVHN, Food-101, ImageNet-R), and remote sensing (Potsdam, LoveDA). The method was tested on diverse backbones, including Swin Transformers, ConvNeXt, ViT, and Mamba-based architectures (VMamba).

ParaX consistently outperformed state-of-the-art PEFT methods like LoRA, AdaptFormer, and Mona:

  • Surpassing Full Fine-Tuning: On the ADE20K semantic segmentation benchmark using a Swin-L backbone, ParaX achieved 52.0% mIoU, surpassing full fine-tuning (51.2%) while utilizing only 3.7% of the trainable parameters.
  • Object Detection: On COCO, ParaX improved upon the previous best method (Mona) by 0.7% APb on Swin-B and significantly outperformed full fine-tuning on ConvNeXt-L.
  • Robustness to Domain Shift: On ImageNet-R, which features significant domain shift, ParaX outperformed Mona by 1.6%, suggesting that dynamic routing helps the model adapt to unfamiliar data distributions.
  • Efficiency: Despite the dynamic routing mechanism, ParaX maintains competitive training speeds and memory usage, matching the footprint of much less accurate baselines.

Limitations and Future Directions

While highly effective, ParaX introduces a few trade-offs. Unlike LoRA, which can be "merged" into the base model weights for zero-latency inference, ParaX weights are dynamic. The router and aggregation logic must run during every inference pass, preventing simple weight merging. Additionally, the use of multi-scale convolutions adds a minor (approximately 15%) overhead to training latency compared to full fine-tuning.

Currently, the study focuses on "Base" and "Large" models. Testing on massive models (e.g., 7B+ parameters) remains a necessary next step. Looking ahead, the authors suggest extending this "Parameters as Experts" philosophy to Large Language Models (LLMs) and Multimodal models to investigate whether dynamic routing can resolve similar redundancy issues in text and cross-modal processing.

Summary updated

Abstract

Adapting pre-trained vision models using parameter-efficient fine-tuning (PEFT) remains challenging, as it aims to achieve performance comparable to full fine-tuning using a minimal number of trainable parameters. When applied to complex dense prediction tasks, existing methods exhibit limitations, including input-agnostic modeling and redundant cross-layer representations. To this end, we propose ParaX, a new adapter-style method featuring a simple mixture-of-experts (MoE) architecture. Specifically, we introduce shared expert centers, where each expert is a trainable parameter matrix. During a feedforward pass, each ParaX module in the network dynamically generates weight matrices tailored for the current module via a simple dynamic parameter routing mechanism, which selectively aggregates parameter matrices in the corresponding expert center. Dynamic weight matrices in ParaX modules facilitate low-rank adaptation in an input-dependent manner, thus generating more customized and powerful feature representations. Moreover, since ParaX modules across multiple network layers share the same expert center, they improve feature diversity by promoting implicit cross-layer feature interaction. Extensive experimental results demonstrate the superiority of ParaX across diverse visual recognition tasks. Code is publicly released at: https://github.com/LMMMEng/ParaX.
BibTeX
@inproceedings{lou2026parameters,
  author = {Meng Lou and Stanley Yu and Yizhou Yu},
  title = {{Parameters as Experts: Adapting Vision Models with Dynamic Parameter Routing}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=EukKoqZhvW},
}
Figure