Routing by Distortion, Not Content: Diagnosing and Fixing a Hidden Expert Collapse in Image Quality Models

Image quality assessment (IQA) models predict how good a picture looks to people. They fail badly when they meet a kind of damage they were not trained on: a metric tuned for blur can mistake JPEG block edges for real detail and score a ruined image as sharp. A natural fix is a mixture of experts (MoE), a network in which a small "router" sends each image patch to one of several specialist sub-networks. In principle the blur patches would go to a blur specialist and the compression patches to a compression specialist. Recent unified quality models already use this design, and the Gamma model built from a CLIP backbone reports a rank correlation of 0.949 on CSIQ (Gamma). Nobody has checked what their routers actually sort by.

We argue that the router in an IQA mixture of experts suffers a form of expert collapse that standard load-balancing cannot see. Because pretrained image features are dominated by semantic content, the router learns to split images by what they show (faces, skies, textures) instead of how they are damaged. Expert usage can then look perfectly balanced while the specialization is useless for unseen distortions. We propose to measure this directly, using the fully crossed design of KADID-10k: 81 reference images, each hit with 25 distortion types at 5 levels, so content and distortion are statistically independent. We then propose a routing objective that trains the router to agree on patches that share a distortion and to disagree on patches that share content but differ in distortion. The test that matters is held-out distortion types. Does routing by distortion improve quality prediction on damage the model has never seen, and does it beat simpler ways of adding the same distortion supervision?

Core Claim

Expert collapse comes in two familiar forms. In starvation, a few experts take all the tokens. In homogenization, aggressive balancing losses make experts redundant (Advancing Expert Specialization for Better MoE; Scaling Vision with Sparse Mixture of Experts). Our claim is that quality models add a third form, which we call factor collapse. Experts can be both busy and different from each other while dividing the input along a dimension that is irrelevant to the task.

This has two consequences. First, balancing methods that only control how much traffic each expert gets cannot repair factor collapse. That includes per-batch losses and the population-level φ-balancing, which tracks a moving average of each expert's historical load (φ-Balancing for Mixture-of-Experts Training). They change how many tokens an expert receives, not which property of a token decides the route. Second, the regularizer should act on routing decisions over controlled pairs of images, not on expert loads or expert outputs. This also explains why simply enforcing orthogonal expert outputs is not enough: experts can be orthogonal yet still keyed to content.

Research Program

The first part of the work is a diagnostic. For any IQA model with a router, we compute each KADID-10k image's routing histogram, meaning the fraction of patch tokens sent to each expert, averaged per layer. We then run a two-way analysis of variance on each expert's usage, with reference image and distortion type as factors. This gives the share of routing variance explained by content, $\eta^2_c$, and the share explained by distortion, $\eta^2_d$. We will apply the diagnostic to four configurations, all on the same CLIP backbone:

  • a reproduction of Gamma's mixture of assessment experts trained only on the quality loss;
  • the same model with a Switch-style per-batch balancing loss;
  • the same model with φ-balancing;
  • the same model with the orthogonality and variance losses of Advancing Expert Specialization for Better MoE.

Gamma's own paper leaves the question of what its router keys on explicitly open. Our first prediction is $\eta^2_c > \eta^2_d$ in all four configurations, including the balanced ones. If instead the router already keys on distortion, the second part of the program is unnecessary, and we would report that clearly.

The second part is the intervention. We borrow Re-IQA's idea of a synthetic distortion bank applied to arbitrary unlabeled images (Re-IQA). For any crop $x$, let $p(x)$ be its image-level routing histogram over the $E$ experts. From the bank we build two kinds of pairs. A pair $(x, x')$ has the same distortion type and level on different content. A pair $(x, x'')$ has the same content with different distortion types. The routing objective is

$$\mathcal{L}_{\text{route}} = \mathbb{E}_{(x,x')}\,\mathrm{JS}\big(p(x),p(x')\big) + \mathbb{E}_{(x,x'')}\,\max\big(0,\; m-\mathrm{JS}(p(x),p(x''))\big),$$

where JS is the Jensen–Shannon divergence and $m$ is a margin. Pairs with the same type at different severity levels are left unconstrained, so experts can be organized by type while each expert's output handles severity.

This objective is added to the quality loss and to φ-balancing. φ-balancing still prevents dead experts, and it is preferable to per-batch losses because distortion clusters within batches are skewed. The bank supplies supervision only for the router. The labeled quality datasets contribute only the usual score loss, so the method does not require distortion labels on authentic photos.

The third part tests a tension raised by On the Adversarial Robustness of Mixture of Experts. That paper found that redundancy among vision experts lets a model survive when perturbations flip expert assignments. Sharper, distortion-keyed routing could lose that protection. We will therefore measure two things under small perturbations: how often routes flip, and how much the predicted score changes. For the perturbations we use both mild synthetic noise and the white-box attacks evaluated by Spectral Robustness Mixer.

Baselines

The naive reference point is the dense CLIP image encoder fine-tuned on the same quality data with the same score loss. It has no experts and no bank pairs. It tells us whether any mixture-of-experts machinery helps at all.

The strongest existing method is Gamma's shared-plus-adaptive expert design (Gamma). Its 0.949 SRCC on CSIQ is the strongest reported number in the file for a mixture-of-experts quality model. We reimplement it on the same backbone, data, and number of training steps. Gamma's prompt component is kept identical across all mixture variants, so only routing differs. We will also report it with each of the three anti-collapse add-ons (Switch loss, φ-balancing, orthogonality plus variance losses). Together these represent the best of the expert-collapse literature applied to this problem. Every variant receives the same hyperparameter search budget, measured in trials over learning rate and loss weight, and is selected on the same validation split.

The non-routing competitor is Re-IQA's two-encoder design (Re-IQA), which separates content from quality without any router. It reports SRCC 0.914 on KonIQ-10k and 0.970 on LIVE. We retrain its regressor on our splits so the comparison is like for like.

The ablation that isolates our key ingredient applies the same bank pairs as a contrastive loss on the dense encoder's features instead of on routing histograms. If this ablation matches our method, the gain comes from distortion-aware supervision, not from routing. A second ablation drops φ-balancing to check whether the routing loss alone causes starvation. An oracle that routes by the true distortion label on KADID gives an upper bound for seen types.

On leave-distortion-out protocols we found no reported result on Lacuna for any method. On PIPAL, the only reported number is SCIQA's SROCC of 0.702 (SCIQA), and that number comes from a full-reference setting that does not match our no-reference protocol. We will therefore count our method as beating the baselines only under three conditions. It must improve held-out-distortion SRCC over the best Gamma variant beyond the spread across three random seeds. It must beat the dense-contrastive ablation. And it must stay within 0.01 SRCC of the best baseline on in-distribution KonIQ-10k and CSIQ.

Why not just…?

Why not just add φ-balancing or a Switch loss to Gamma? That is exactly our second through fourth baselines. Our hypothesis is that they fix the amount of traffic each expert gets but not its basis. A balanced router that splits by content would show near-uniform loads and a high $\eta^2_c$. If balancing alone raises $\eta^2_d$ and held-out SRCC as much as our method does, the routing objective is unnecessary, and the diagnostic finding still stands as the contribution.

Why not just train a dense model with the same distortion bank? This is the most serious threat. Distortion-aware pretraining is what makes Re-IQA strong without any router. The dense-contrastive ablation tests this directly with identical pairs and compute. We consider the threat real. A routing-specific benefit should appear mainly on held-out distortion types and on images that mix several kinds of damage, because there patch-level routing can send different regions to different experts and a single dense pathway cannot. If no such gap appears, we will conclude that routing is not the mechanism.

Why not just use Soft MoE, which cannot collapse by construction? Soft MoE beats token-choice routing on clean vision benchmarks, for example 67.85% versus 65.91% in 10-shot ImageNet (Routers in Vision Mixture of Experts). But it removes starvation, not factor collapse. Its experts see blends of all tokens, which may dilute local artifacts such as ringing near a single edge. We include it as a baseline and run the same variance diagnostic on its mixing weights.

Why not just route by a distortion classifier? On KADID, the oracle-label router shows what that approach achieves. A classifier trained on a fixed list of distortion types has no category for the unseen ones, and those are precisely the held-out test.

Proposed Experiments

All models are trained on KADID-10k plus KonIQ-10k. The bank pairs use a 25-type distortion bank applied to unlabeled images.

The decisive experiment is leave-distortion-out on KADID. We hold out five types, for example all compression-family types, retrain, and test on them. This is repeated for three disjoint held-out families. Further out-of-distribution transfer uses PIPAL, whose GAN-generated distortions are absent from the bank, and exBeDDE dehazing. Metrics are SRCC and PLCC.

We also report the correlation-surface analysis of From Global to Granular. That paper showed that a single correlation number can hide failures confined to particular quality bands, so we check whether the gains are concentrated in the low-quality or high-quality regime.

Alongside accuracy, we report three mechanism measures for every model: $\eta^2_c$ and $\eta^2_d$ per layer, the number of dead experts, and the route-flip rate under perturbation.

The core claim is supported if three things hold together. First, $\eta^2_d$ rises substantially under our objective while the balancing baselines leave it low. Second, the rise in $\eta^2_d$ tracks held-out SRCC across variants. Third, our method beats the dense-contrastive ablation on held-out types.

Two outcomes would falsify the claim. The first is that balancing alone produces distortion-keyed routing. The second is that distortion-keyed routing raises $\eta^2_d$ but leaves held-out SRCC unchanged.

A further outcome, a sharp increase in score sensitivity when routes flip, would be informative on its own. It would mean that the redundancy paradox constrains how far IQA experts should be pushed apart.