Large vision-language models such as CLIP (a model trained to match images with captions) have become a popular starting point for predicting how good a photo looks to people. But they carry a known bias: they tend to score an image by what it shows rather than how well it was captured, so a blurry dog can look "high quality" simply because it is a dog. Current adaptation methods add small learned "prompt" vectors or low-rank weight updates to CLIP. These additions are identical for every image, and they struggle on real-world photos, where noise, blur and bad exposure mix in continuous and unpredictable ways. Making the adaptation depend on the input is the obvious fix, and it has worked well outside quality assessment. The catch is that input-dependent adapters usually take their conditioning signal from the backbone's own features. In CLIP those features are dominated by content, so a naive dynamic adapter risks learning "portraits get prompt A, landscapes get prompt B," which amplifies exactly the bias we want to remove.
This note proposes to drive the adapter from a signal that cannot see content: cheap, hand-computed local image statistics of the kind classical no-reference quality metrics use. These include contrast-normalized pixel statistics that capture departures from natural-scene regularities, and patch-level frequency energy. A small network turns these per-patch statistics into per-token gates over the directions of a low-rank adapter inside a frozen CLIP. The claim is that the adapter should vary with degradation, not with content. If it does, a content-blind gating signal should beat both static adapters and self-conditioned dynamic adapters on in-the-wild and cross-dataset quality prediction, and it should reduce semantic shortcut errors. All three effects can be measured directly.
What exists and where this sits
Three papers already adapt CLIP for quality assessment with learned prompts, and all three leave the adaptation static. Multi-Modal Prompt Learning on Blind Image Quality Assessment learns separate scene and distortion text prompts plus deep visual prompt tokens, reaching SRCC 0.961 on CSIQ and 0.941 on KADID. Its authors report that gains shrink on KonIQ-10k because fixed prompt categories are too coarse for compound real-world degradations. That observation motivates this proposal. Boosting CLIP Adaptation for Image Quality Assessment via Meta-Prompt Learning and Gradient Regularization meta-learns a prompt initialization and projects out the part of each gradient update that aligns with semantic recognition. It cuts semantically misled samples by 73% but deploys fixed prompts at test time. We attack the same bias architecturally, at inference, rather than through the optimizer, and we will compare against it directly. Bringing Textual Prompt to AI-Generated Image Quality Assessment uses a static learned [QA] token and conditions on the user's generation prompt. That is a different sense of "prompt," so the work bears on this proposal only as an evaluation setting (AGIQA).
The mechanism comes from the input-dependent adaptation literature. Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating gates each low-rank direction per token and stabilizes the gates with a moving-average prior. On LLaMA-3-8B math tasks, rank 8 with gating reaches 76.02% against 71.67% for static rank 32. We reuse its gating form and its stabilizer. Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts generates visual prompts from the image's own features. It holds 60.1% on Stanford Dogs with 1% of the data, where static prompts collapse to 3.6%. It is our main self-conditioned competitor. Hystar conditions on a separate DINOv2 style embedding and modulates only attention singular values, keeping MLP layers static. It found that making everything dynamic destabilized training. That constraint shapes our design, and the paper supplies a precedent for conditioning on a signal external to the adapted backbone. Our conditioning signal is different in kind. It is not a learned embedding at all but deliberately content-poor statistics, borrowed from Blind image quality assessment based on perceptual features and natural scene statistics. That paper feeds such statistics into a shallow support vector regressor, whereas we use them only to route the backbone's adaptation.
Core Claim
For frozen CLIP adapted to predict mean opinion scores, gating low-rank adapter directions by local image statistics beats three alternatives on authentic and cross-dataset benchmarks: gating by CLIP's own features, a static adapter of larger rank, and appending the same statistics to the regression head. The gains should be concentrated on content-confounded images. The learned gates should be explained mostly by distortion type rather than by content, which a self-conditioned gate will not show.
Research Program
The model is a frozen CLIP ViT-B/16 image encoder. Each attention projection gets a rank-8 adapter $\Delta W_t = B\,\mathrm{diag}(g_t)\,A$. Here $A$ and $B$ are the learned low-rank factors, and $g_t \in [0,1]^8$ is a gate vector for token $t$ produced by a two-layer network from $s_t$, a roughly 40-dimensional statistics vector for that token's patch. The vector $s_t$ holds contrast-normalized coefficient moments and fitted shape parameters at two scales, plus band-wise frequency energy. Following the gating paper, a sequence-level average of $s_t$ is mixed into each gate to prevent erratic token-to-token behavior. A bias-corrected moving-average prior regularizes the gates. Following Hystar, MLP blocks keep ordinary static adapters.
For mixed-dataset training, the regression head is the monotonic, dataset-specific mapping of Learning from Mixed Datasets: A Monotonic Image Quality Assessment Model. It maps a shared scalar score through positive-weight layers per dataset, so datasets with different rating scales do not fight over the score. The statistics cost a few convolutions per image. The adapters can no longer be merged into the weights, so we will report measured latency against static LoRA and not claim this away.
The analysis half matters as much as the scores. On KADID-10k, every reference image appears with 25 distortion types at five levels. This lets us decompose gate variance into distortion-type, level, and content (reference-image) components for each conditioning signal. The hypothesis predicts a high distortion share for statistics-gating and a high content share for self-gating.
Baselines
The naive reference point is frozen CLIP with an MLP regression head and no adaptation. It shows how much any adaptation buys, and per the GRMP-IQA analysis it should exhibit the strongest semantic bias. The static reference is ordinary LoRA at rank 8 and rank 32 in the same layers, trained with the same monotonic head. Rank 32 is included because "just add capacity" is the cheapest objection.
The strongest published methods on the data-efficient protocol are GRMP-IQA, which reports SRCC 0.853 on KonIQ-10k and 0.836 on LIVEC using 20% of training data, and MP-IQE, which reports 0.961 on CSIQ and 0.941 on KADID. We found no reported KonIQ-10k number for MP-IQE on Lacuna. With full training data, MUSIQ reports SRCC 0.916 on KonIQ-10k and is the full-data reference. We will rerun GRMP-IQA and MP-IQE on our exact splits with the same CLIP backbone where code permits, and quote published numbers only alongside, not in place of, reruns. Beating GRMP-IQA means higher SRCC on KonIQ-10k and LIVEC at 20% data, averaged over ten splits, with the gap larger than the split-to-split standard deviation.
Two ablations isolate the key ingredient. The first is self-conditioning: the identical gated adapter with gates computed from CLIP's own patch tokens, plus VAPT run at its published configuration. The second is late fusion: static LoRA with the statistics vector concatenated into the regression head. Every method gets the same trainable-parameter budget within 10%, the same epochs, and the same 20-trial hyperparameter search on a held-out validation split.
Why not just…?
Why not just use an existing dynamic prompt method like VAPT? This is the most direct port. It may well beat static prompts, given its low-data robustness on classification. But VAPT conditions on the backbone's own features, and in CLIP those are content-heavy. So we expect it to fix rigidity while preserving or worsening semantic bias. The self-conditioned ablation and the gate-variance decomposition settle this. If self-conditioning matches statistics-gating on both SRCC and the bias probe, our central claim is wrong, and we will report it as such.
Why not just raise the static rank or train longer prompts? Evidence outside quality assessment says rank does not substitute for input dependence: gated rank 8 beats static rank 32 in the U-LoRA results. Quality regression is a different target, though, so the rank-32 baseline is run rather than argued.
Why not just append the image statistics to the regression head? This is the real threat. Classical support vector regression on the same statistics already works on synthetic data, and a concatenated head is simpler and mergeable. The argument against it is that a head cannot change which features CLIP computes. If CLIP's features are content-dominated, the head can only reweight them. Late fusion is therefore a required ablation. If it matches statistics-gating, the recommendation becomes "use late fusion," and the dynamic adapter is unnecessary.
Why not a classifier that picks among a dictionary of distortion prompts? This is essentially MP-IQE's discrete design, whose own authors report weaker gains on authentic data. We include a 25-entry dictionary keyed by a KADID distortion classifier to confirm the effect under our protocol.
Proposed Experiments
We train and test within dataset on KonIQ-10k, LIVEC, SPAQ, CSIQ and KADID-10k, using both 20% and full training splits, and report SRCC and Pearson correlation. The decisive generalization test trains on KADID-10k and TID2013 (synthetic distortions) and tests on KonIQ-10k, LIVEC and BID (authentic). This is where compound real-world degradations should separate continuous statistics-gating from static and discrete prompts.
The bias probe reimplements GRMP-IQA's count of semantically misled samples. We add a content-controlled check: on KADID, we compute rank correlation within each reference image (content fixed, distortion varies) and within each distortion level (distortion fixed, content varies). A content-blind gate should raise the first without raising the second.
Two stress tests probe where content-blind statistics may fail. PIPAL's GAN-restored images may look statistically natural while being wrong, and PIPAL's reported LPIPS-Alex correlation on GAN distortions is only 0.4862 (0.5135 with blur pooling). AGIQA-1k is the second stress test, with IP-IQA's reported 0.8401 SRCC as reference. A loss there would bound the method's scope honestly, not refute the core claim.
Latency and FLOPs are reported against static LoRA at 512-pixel inputs. The project succeeds if statistics-gating beats self-gating, late fusion and rank-32 LoRA on the synthetic-to-authentic transfer, and if its gate variance is dominated by distortion rather than content.
References
- Multi-Modal Prompt Learning on Blind Image Quality Assessment
- Boosting CLIP Adaptation for Image Quality Assessment via Meta-Prompt Learning and Gradient Regularization
- Bringing Textual Prompt to AI-Generated Image Quality Assessment
- Adaptive Utilization of Low-Rank Adaptation via Conditioned Gating
- Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts
- Hystar: Hypernetwork-driven Style-adaptive Retrieval via Dynamic SVD Modulation
- Blind image quality assessment based on perceptual features and natural scene statistics
- Learning from mixed datasets: A monotonic image quality assessment model