TL;DR
RouteMark is a novel fingerprinting framework designed to protect intellectual property (IP) in merged Mixture-of-Experts (MoE) models. By analyzing the unique, task-specific routing behaviors of individual experts rather than their raw weights or activations, RouteMark can reliably identify stolen or reused components even after significant tampering like pruning, fine-tuning, or structural modification.
What Problem Does This Paper Address?
As large-scale AI models become more specialized, "model merging" has emerged as a cost-effective way to combine multiple fine-tuned models into a single, multi-task system. The Mixture-of-Experts (MoE) architecture is the gold standard for this, using a "router" to dynamically select which specialized sub-module (expert) should process a given input.
However, this modularity creates a massive IP loophole. An adversary can easily "borrow" a high-performing expert from a victim's model and plug it into their own merged system. Traditional IP protection methods fail here:
- Weight-based methods are easily defeated by simple parameter permutations or pruning.
- Activation-based methods are often too coarse, making it hard to pinpoint which specific expert in a large pool was reused.
- Watermarking often doesn't survive the merging process itself.
The authors address the need for a "fine-grained" attribution method that can identify specific experts within a merged model, even if the model's structure or parameters have been altered.
Methods
The core innovation of RouteMark is shifting the focus from what an expert is (weights) to how an expert is used (routing behavior). The authors observe that experts retain a "behavioral signature" tied to their original training task.
RouteMark constructs two complementary fingerprints for each expert based on routing logits collected during inference on a small set of "probe" inputs:
- Routing Score Fingerprint (RSF): A matrix that captures the intensity of an expert's activation across different tasks and layers. It uses expert-wise mean-centering to remove layer-specific biases, leaving a clean signature of activation intensity.
- Routing Preference Fingerprint (RPF): A probability vector that summarizes an expert's task-level specialization. It identifies which types of inputs the router "prefers" to send to that specific expert.
To verify ownership, the framework uses a similarity-based matching algorithm. It compares the fingerprints of a "suspect" expert against a "victim" expert using cosine similarity (for RSF) and Jensen-Shannon divergence (for RPF).
Datasets
The authors evaluated RouteMark using CLIP-based MoE models merged from experts fine-tuned on eight diverse vision datasets:
- Natural Scenes: SUN397
- Objects: Cars
- Aerial/Satellite Imagery: RESISC45, EuroSAT
- Digits: SVHN, MNIST
- Traffic Signs: GTSRB
- Textures: DTD
These datasets provide a rigorous testbed for expert specialization across heterogeneous visual domains.
Key Insights and Ideas
- Routing as a Behavioral Proxy: The most striking insight is that routing behavior is more stable than weights. Even if an adversary fine-tunes or prunes an expert, the router's fundamental logic for selecting that expert remains tied to the expert's original domain.
- Permutation Invariance: Unlike weight-based fingerprints, RouteMark is naturally invariant to parameter reordering. If an adversary shuffles the internal dimensions of an expert to hide its identity, the routing logits remain unchanged, and the fingerprint stays intact.
- The "Aha" Moment on Sample Efficiency: Surprisingly, RouteMark becomes more stable with fewer samples compared to activation-based methods. While activation-based methods (like REEF) get noisier as more experts influence the representation, routing logits provide a direct, cleaner signal of expert utility.
Key Results
- High Discriminability: Reused experts consistently showed similarity scores above 0.96, while unrelated experts stayed below 0.20, providing a massive margin for clear attribution.
- Robustness to Pruning: In tests using the WANDA pruning method, RouteMark maintained a similarity score above 0.9 even at 50% sparsity. In contrast, the activation-based baseline (REEF) plummeted to 0.4.
- Structural Tampering Resilience: The method remained stable even when the model structure was modified by adding, deleting, or replacing up to three experts.
- Superiority over Baselines: RouteMark consistently outperformed weight-based (PCS, ICS) and activation-based (REEF) methods across all tampering scenarios, often by margins of 10-20%.
Limitations and Future Work
While highly effective, the authors note a few areas for further exploration:
- Architecture Scope: The current study focuses on CLIP-based vision models. While the logic should apply to Large Language Models (LLMs), empirical validation on transformer-based text MoEs is a logical next step.
- Scaling: Future work could investigate how RouteMark performs in "Massive MoE" scenarios with hundreds or thousands of experts, where routing collisions might become more frequent.
- Multimodal Merging: Extending the framework to handle models that merge experts across different modalities (e.g., vision and audio) remains an open challenge.
Summary updated