Image quality models treat blur, grain, darkness and distortion as damage. Photographers, painters and image generators often use exactly these effects on purpose. A creamy out-of-focus background behind a sharp face, film grain requested in a text prompt, and low-key lighting in a portrait all have the same local pixel statistics as a camera failure. The literature documents how badly this goes. A leading vision-only quality model scores stylized grainy AI images with a Spearman rank correlation (SRCC) of 0.2545 against human ratings, and adding a large language-vision model's judgment recovers this to 0.8364 (Large Multi-modality Model Assisted AI-Generated Image Quality Assessment). That repair works on a global vector, though. It cannot say which degradation was forgiven or why.
We propose a quality model that makes this decision explicit and local. A frozen degradation estimator first measures how much blur, noise or distortion each image patch contains. A small bidirectional attention module then reads the patches together with a set of text "intent hypotheses" such as "shallow depth of field portrait," "deliberate film grain," "chiaroscuro," or the generation prompt itself. It outputs, for every patch, a gate between 0 and 1 that says how much of that patch's measured degradation is excused as style. The final score charges only the unexcused degradation. Because the degradation estimate is fixed, the gate cannot invent quality; it can only forgive damage that was actually measured. The result is a spatial "excuse map" that can be checked against pixel-level labels of good and bad blur.
Core Claim
Aesthetic-technical ambiguity should be resolved per region and conditioned on a stated or inferred style, not by a global fusion weight. We make two testable claims. First, a two-way attention step matters. In the first direction the image picks and reshapes which intent hypothesis applies. In the second the chosen hypothesis decides which patches' degradation to forgive. This two-way step beats one-way text-queries-image attention, and it beats global gating, on cases where intent is not spatially separable, such as whole-frame grain, low-key lighting, or motion blur as style. Second, on spatially separable cases like bokeh, a purely visual subject-saliency rule may do just as well. We expect this, and the evaluation is split so that it shows up plainly rather than being averaged away.
Concretely, let $d_i \ge 0$ be the frozen estimator's degradation magnitude for patch $i$, $w_i$ a learned perceptual weight for that patch, $a$ an aesthetic score from the whole image, and $g_i \in [0,1]$ the excuse gate. The predicted quality is
$$q = a - \sum_i w_i \,(1 - g_i)\, d_i .$$
The gate $g_i$ is computed from patch $i$'s visual token and the intent context it retrieves through cross-attention. The perceptual weight plays the same role as the salience weighting in TempQT (Blind Image Quality Assessment via Transformer Predicted Error Map and Perceptual Quality Token). The difference is that TempQT asks "does this region matter?" while the gate asks the separate question "was this degradation meant?"
Relation to Prior Work
Several lines of work supply parts of the method. The closest vision-language quality model, MP-IQE (Multi-Modal Prompt Learning on Blind Image Quality Assessment), makes CLIP (a contrastively pre-trained image-text encoder) quality-aware. It does this by letting learned scene and distortion prompts query the image through three cross-attention layers. The flow runs one way only: the prompts never adapt to the image, and it trains only on photographic distortion datasets. The authors also note weaker gains on the diverse KonIQ-10k. LIQE (Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective) scores quality by global cosine similarity against 495 fixed scene-distortion-grade sentences, so it has no patch-level reasoning. AesPrompt (AesPrompt: Zero-Shot Image Aesthetics Assessment With Multi-Granularity Aesthetic Prompt Learning) and ArtCLIP (APDDv2: Aesthetics of Paintings and Drawings Dataset with Artist Labeled Scores and Comments) bring rich aesthetic language and expert attributes, but they also match global embeddings. AesPrompt's authors report that chiaroscuro, deliberate motion blur and unusual framing are still misread as defects. These papers motivate the proposal: they show language helps, and that static, global language falls short exactly on intentional deviations.
MA-AGIQA (Large Multi-modality Model Assisted AI-Generated Image Quality Assessment) is the strongest existing answer to the problem. It fuses low-level MANIQA features with a frozen multimodal language model's semantic features through a mixture-of-experts gate. We differ in where the decision is made. MA-AGIQA decides once per image, while our gate decides per patch and is tied to a measured degradation. DOVER (Disentangling Aesthetic and Technical Effects for Video Quality Assessment of User Generated Content) separates technical and aesthetic quality but combines them with a fixed ratio. That fixed ratio is precisely what fails when distortion is the medium.
The mechanism is borrowed from two places. CALIP (CALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention) shows that in CLIP's shared space, text features can be refined by image patches and patches by text using plain dot products. It was tested only on classification. The gated cross-attention of An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention mixes cross-modal context with each token's original features through a learned gate and uses sparsemax, a softmax variant that can assign exact zeros. It shows this stops a large modality from washing out a small one. We reuse both ideas, but give the gate a meaning it lacked there: the fraction of measured damage excused.
Vision-only multiscale models constrain the design. BiRQA (BiRQA: Bidirectional Robust Quality Assessment for Images) passes information both up and down a visual pyramid and separates texture from compression artifacts. PaQ-2-PiQ (From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality) concedes that its local-to-global feedback still sometimes penalizes bokeh. These results tell us that visual context alone is not enough in general. They do not tell us it is not enough for bokeh, and we test that directly.
Research Program
The backbone is CLIP ViT-B/16 as used by LIQE, built from the public code (zwx8981/LIQE). That backbone gives 196 patch tokens. The degradation estimator is a patch-level technical branch trained only on synthetic distortions (KADID-10k, CSIQ) and then frozen. Its per-patch outputs $d_i$ therefore mean "how damaged does this look," with no knowledge of style. The intent side is a bank of about 32 learned hypothesis prompts, seeded from descriptions of photographic and painterly techniques. For AGIQA-3k images, the generation prompt is added as an extra, declared-intent sequence.
The module runs one round of two-way attention. In the first half, hypotheses attend over patches, producing image-conditioned hypotheses. A sparsemax over hypotheses then selects the few that plausibly apply, which limits the damage from irrelevant or hallucinated text. In the second half, each patch attends over the selected hypotheses and receives a context vector. The gate $g_i$ comes from a small network on the patch token and this context, and the visual token is updated with the residual gate so its identity is preserved. We cap interaction at one or two rounds, because repeated message passing is known to blur the two modalities together. With 196 patches and at most 32 hypotheses, the quadratic cost is negligible.
Training uses mean opinion scores from KonIQ-10k, AGIQA-3k and APDDv2, plus SmartBlur's image-level good/OK/bad blur labels as an auxiliary classification loss. SmartBlur's pixel-level blur maps are held out from training and used only to evaluate the excuse maps.
Baselines
The naive reference is the frozen degradation estimator with no gate. Its score is simply $a - \sum_i w_i d_i$, and it penalizes every blur and grain. We also report LIQE run from its official code. This anchors how much any intent mechanism buys.
The strongest existing method on the key question is MA-AGIQA. On AGIQA-3k it reports overall SRCC 0.8939, and 0.8364 on its artistic grainy subset against MANIQA's 0.2545 (Large Multi-modality Model Assisted AI-Generated Image Quality Assessment). It is the strongest competitor because it already injects language-model world knowledge. If a global gate over that knowledge suffices, our per-patch machinery is unnecessary. We will retrain it on our training mixture with the same splits and the same hyperparameter search budget. If the exact grainy subset is not released, we will rebuild a subset by the same criterion and rerun both MANIQA and MA-AGIQA on it rather than quoting their numbers. Beating it means matching its overall SRCC on AGIQA-3k while exceeding it on the grainy and global-style subsets. It also means producing excuse maps that it cannot produce at all.
For one-way attention, the baseline is MP-IQE. It reports SRCC 0.961 on CSIQ and 0.941 on KADID-10k, and we found no reported result on Lacuna for MP-IQE on KonIQ-10k, AGIQA-3k or APDDv2 (Multi-Modal Prompt Learning on Blind Image Quality Assessment). We reimplement it on the same backbone and training data.
For blur desirability, the reference is ABC-FuseNet at 81.4% accuracy on SmartBlur (Learning to Understand Image Blur). For paintings, it is ArtCLIP at SROCC 0.81 on the APDDv2 total aesthetic score (APDDv2: Aesthetics of Paintings and Drawings Dataset with Artist Labeled Scores and Comments).
Three ablations isolate the ingredients, all with the same parameter count where possible. The first removes the image-to-text half, so the hypotheses stay fixed and only query patches. The second removes the gate and adds context by plain residual. The third replaces sparsemax with softmax.
A visual-only saliency gate is a further baseline, and it is the one we take most seriously. In it, $g_i$ is set from a subject-saliency map and the local blur map in the style of ABC-FuseNet, with no text at all.
Why not just…?
Why not just fine-tune MA-AGIQA or a multimodal language model end to end? This is the most serious threat to the aggregate numbers. A fine-tuned large model may well match us on overall SRCC. Its evidence, however, is global: the grainy-subset recovery is reported per image, with no account of where in the image the forgiveness applies. Our claim that locality matters is falsifiable in two ways. The excuse maps can be scored against SmartBlur pixel labels. And on mixed images, with intentional bokeh plus an unintended shake on the subject, a global gate must forgive or penalize both, whereas a local gate can separate them. If MA-AGIQA matches us on those mixed cases, the per-patch claim fails.
Why not just use subject saliency, since blur outside the subject is usually intended? For bokeh this may be enough, and we expect the visual-only saliency baseline to come close to us on SmartBlur's good-blur class. It cannot work when the intended degradation covers the subject: whole-frame film grain, low-key lighting on a face, or motion blur across a moving figure. The evidence that vision alone fails here is MANIQA's 0.2545 on grainy AI art. We report bokeh-like and global-style subsets separately. If saliency ties us on both, the language channel is not earning its cost, and we would drop it.
Why not just condition the prompts on the image and keep one-way attention? Image-conditioned prompts followed by text-queries-image attention are close to our design. The first ablation, which removes the image-to-text half, is this alternative at matched size. The two-way design is justified only if it wins on global-style cases and on prompt-swap sensitivity, where the hypothesis must first be chosen from the image before it can direct forgiveness.
Why not just apply CALIP's parameter-free attention to LIQE? It costs nothing, so we run it zero-shot as a floor. CALIP was shown only on categorical recognition, and it has no gate tying context to a measured degradation. We expect it to shift scores toward whatever text matches best without distinguishing forgiven from unforgiven damage. The excuse-map evaluation will show whether that expectation holds.
Proposed Experiments
We report SRCC and PLCC (Pearson linear correlation) on KonIQ-10k, AGIQA-3k, APDDv2 and CSIQ. We also report accuracy on SmartBlur's three blur classes. Any drop against MP-IQE on CSIQ or KonIQ-10k counts as a cost of the method; forgiving style must not come at the price of photographic accuracy.
Three checks decide the claim. The first is localization. On held-out SmartBlur images, the excuse map should overlap good-blur pixels and avoid bad-blur pixels, measured by area under the ROC curve against the pixel labels. The saliency gate and MA-AGIQA (via gradient attribution) serve as comparisons.
The second is a split evaluation. We take AGIQA-3k and APDDv2 images and partition them into spatially separable intent (background blur) and global-style intent (grain, low-key lighting, painterly texture). Annotators will label these, using the generation prompt where available. The proposal succeeds only if it beats the saliency gate on the global-style split and beats MA-AGIQA on at least one split.
The third is a counterfactual prompt swap. On AGIQA-3k we replace each generation prompt with a style-neutral one. A model that uses declared intent should increase its penalty on grainy or blurred images and leave clean images unchanged. A small human study on about a thousand swapped pairs checks whether that shift matches how people judge the image when told what was asked for.
If the gate turns out to fire uniformly, or to track generic image appeal rather than measured degradation, the decomposition has failed. We would then report it as a negative result on per-region intent modeling.
References
- Large Multi-modality Model Assisted AI-Generated Image Quality Assessment
- Blind Image Quality Assessment via Transformer Predicted Error Map and Perceptual Quality Token
- Multi-Modal Prompt Learning on Blind Image Quality Assessment
- Blind Image Quality Assessment via Vision-Language Correspondence: A Multitask Learning Perspective
- AesPrompt: Zero-Shot Image Aesthetics Assessment With Multi-Granularity Aesthetic Prompt Learning
- APDDv2: Aesthetics of Paintings and Drawings Dataset with Artist Labeled Scores and Comments
- Disentangling Aesthetic and Technical Effects for Video Quality Assessment of User Generated Content
- CALIP: Zero-Shot Enhancement of CLIP with Parameter-Free Attention
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- BiRQA: Bidirectional Robust Quality Assessment for Images
- From Patches to Pictures (PaQ-2-PiQ): Mapping the Perceptual Space of Picture Quality
- Learning to Understand Image Blur