Do Aesthetic Explanations Move When the Image Does?

Models that judge how good a photo looks increasingly explain themselves. They output per-attribute ratings ("lighting 8, composition 4") or written critiques, and they are ranked almost entirely by how well their overall score correlates with crowd ratings. Nobody checks whether the explanation tracks the image. If we darken a photo, does the model's "lighting" rating drop? If we recrop so the subject leaves the rule-of-thirds line, does its composition critique change? And does its overall score change by roughly what its own attribute ratings imply? This proposal builds a locked evaluation harness that answers these questions. It applies controlled, human-validated edits to images from AVA, AADB, and PARA and measures whether each model's stated attributes and critiques respond in the right direction.

The claim is that correlation with mean opinion scores and faithfulness of explanations are separable properties. Under a single fixed protocol, the two will rank current interpretable aesthetic models differently. The faithfulness measure, not the correlation measure, will better predict how much a model degrades on out-of-distribution datasets. The idea borrows the logic of reasoning-shortcut audits in rsbench. That work showed models with around 80% task accuracy but only 10–20% concept accuracy, and an average 46% drop under distribution shift. We carry this into visual aesthetics, where the "concepts" are photographic attributes. We ground them with image interventions rather than synthetic labels.

Why this gap is real

Aesthetic assessment has moved from single scores toward interpretable outputs along three lines. The first is attribute networks: Kong et al.'s ranking network predicts 11 AADB attributes, and AGREE routes five low-level attributes (brightness, contrast, blur, hue, saturation) through separate adapters to avoid gradient conflict. The second is critique generators such as AMAN. The third is multimodal language models that reason before scoring, such as UNIAA-LLaVA and Aes-R1. All three lines are evaluated mainly on overall correlation (SRCC and PLCC), text-overlap scores, or multiple-choice perception accuracy.

Two findings motivate the proposal directly.

  • Composition. UNIAA reports that composition perception accuracy stayed below 60% for every model it tested, even while overall scores were competitive.
  • Critique grounding. AesCanvas found that fluent critiques often miss decisive visual evidence. ArtiMuse fell from 21% raw accuracy to 3% once correct answers had to cite the right visual cue.

Aes-R1 filters its reasoning data for score leakage but never tests whether its generated reasoning is causally tied to pixels. AGREE derives its routing from programmatic perturbations but does not check that the routed attributes match human perception.

The second motivation is protocol noise. NerfBaselines showed that undocumented downscaling and bit-depth handling flipped leaderboards. SimpleView showed that porting a newer training protocol lifted PointNet++ from 90.0% to 93.3% on ModelNet40, erasing years of claimed architectural gains. The study of MLLMs for image quality showed that prompting setup alone moves GPT-4V's SRCC on SPAQ from 0.860 to 0.881.

Aesthetic papers report gains of a similar size, such as AGREE's 1.6–4.8% relative correlation improvement. They do so under dataset-specific preprocessing and score-extraction choices. Before faithfulness can be compared across models, the protocol has to be frozen and its own variance measured.

The closest existing work is UNIAA-Bench and AesCanvas. UNIAA-Bench unifies perception, description, and scoring into one benchmark, but its signals are multiple-choice answers and correlations; none of them intervene on the image. AesCanvas's evidence-grounded accuracy checks whether a critique cites the right cue. However, it rests on 301 expert-curated cases, targets contextual suitability rather than attribute faithfulness, and ships no harness that fixes prompts or decoding. VHELM provides that kind of locked harness for 22 vision-language models but contains no aesthetic scenarios. We take VHELM's locked-protocol approach and AesCanvas's insistence on grounding, and add what neither has: automatic, scalable, human-validated counterfactual image edits.

Research Program

The work has three parts.

1. A frozen harness. Every model is evaluated through one pipeline. It fixes:

  • image resizing and color handling;
  • the official splits of AVA, AADB, TAD66K and PARA, where they exist, plus published random-seed splits where they do not;
  • for language models, one prompt template per task, greedy decoding, and a single score-extraction rule. Scores are read as expected values over rating-token probabilities where logits are available, and parsed from text otherwise. Both rules are reported for models that allow both.

Before any faithfulness result is reported, we run a variance study. We perturb each of these choices one at a time: the resize policy, the split, the prompt wording across five paraphrases, and the extraction rule. We then report the spread in SRCC per model. This tells us which published differences between models exceed protocol noise. It also serves as the "do-nothing" reference for everything that follows.

2. An intervention set with human ground truth.

  • Low-level attributes. For each of AGREE's five attributes, we apply graded edits to held-out AVA, AADB and PARA images at three strengths in both directions.
  • Composition. We generate recrops of the same image that move the main subject onto or off rule-of-thirds points and diagonals. The candidate crops are parameterized by the explicit composition rules of CACNet.

We do not assume the direction of a perceptual effect. For example, extra blur can improve a portrait with intentional background softness, which is exactly the conflict AGREE documents. Instead, we collect paired human judgments on a subset of roughly two to three thousand original/edited pairs. Raters say which version has better lighting, sharpness, composition and so on, and which is better overall. Human agreement on each pair sets an upper bound. Only edits whose attribute direction reaches clear human majority enter the scored set, and a separate "ambiguous" slice is kept for analysis.

3. Faithfulness measures. We define three.

  • Attribute-direction consistency: the fraction of validated edits for which the model's stated attribute moves the same way as the human majority.
  • Critique consistency: for critique-producing models, whether the critique's claim about the edited attribute flips. This is scored by matching claims against a fixed attribute lexicon. A secondary language-model judge is used only for claims the lexicon cannot resolve, and that judge is audited against human labels on a sample.
  • Score mediation: whether a model's change in overall score is predicted by its own attribute changes about as well as human overall preference is predicted by human attribute judgments.

A model that fluently critiques lighting but whose overall score ignores lighting edits fails the third measure even if it passes the first. That pattern is the aesthetic analogue of the reasoning shortcut rsbench describes.

We evaluate models where weights or code are available: Kong et al.'s attribute ranking network, AGREE on its reported backbones, UNIAA-LLaVA, Qwen2.5-VL-7B both base and with the Aes-R1 recipe, and one or two closed multimodal models under the same prompt. If a model is unavailable, we reproduce its training recipe inside the harness and label it as a reproduction.

Baselines

The measurement being tested here is an evaluation criterion, so the baselines are competing ways to rank and diagnose models, plus the models themselves as reference points.

Do nothing. The naive reference is each paper's self-reported numbers under its own protocol. Examples are Kong et al.'s SRCC of 0.6782 on AADB against an average human rater consistency of 0.6738, and UNIAA-LLaVA's 78.48% on UNIAA-QA against 74.74% for GPT-4V. We re-run each model in the frozen harness and report how far its correlation moves from the published value. We also report whether pairwise orderings survive once the protocol-variance band from the first part is applied.

Strongest existing evaluation signal. This is overall SRCC/PLCC in the frozen harness, because the field uses it to choose between models. The strongest scoring method on this signal in the file is Aes-R1. It reports relative improvements of 47.9% in PLCC and 34.8% in SRCC over its Qwen2.5-VL-7B starting point, with PARA and AADB used for out-of-distribution testing. AGREE reports 1.6–4.8% relative correlation gains and 17.6–38.5% gains on hard samples. No absolute Aes-R1 SRCC on PARA is given in the file, so we will report what the harness measures rather than quote a target.

"Beating" this baseline has a specific meaning. We train on AVA under the frozen protocol and measure each model's drop to PARA and AADB. Across all evaluated models, intervention consistency should rank-correlate with that drop more strongly than in-distribution SRCC does. The model sample is small, so the difference will be tested by bootstrap over images and models, and we will say plainly if the difference is not significant.

Grounded-critique baseline. This is an AesCanvas-style evidence-grounded accuracy check run on our own critiques. Human annotators mark whether the critique names the decisive cue on a subset of unedited images. It is the strongest existing notion of explanation grounding, and it is scored on the same images and models.

Key ablation. We drop the human validation and score edits against the direction assumed by the programmatic perturbation, which is how AGREE's sensitivity routing treats them. If the validated and unvalidated versions rank models identically, the costly human step is unnecessary, and we will say so.

Why not just…?

Why not just build a clean unified harness and report correlation with more seeds? That is the first part of this proposal, and it is necessary: SimpleView and NerfBaselines show that protocol cleanup alone can reverse conclusions. But it cannot say whether an explanation is faithful. A model can achieve high correlation while its "composition" output is decorative, which is what UNIAA's sub-60% composition accuracy alongside competitive scores suggests. The decisive check is whether faithfulness adds predictive power for out-of-distribution drops beyond clean SRCC. If it does not, the intervention machinery is not worth its cost, and the clean harness is the real contribution.

Why not just grade critiques with a language-model judge? VHELM relies on Prometheus-Vision without auditing its behavior on subjective criteria. AesCanvas shows that fluent critiques routinely hide missing evidence, and a text-reading judge is exposed to exactly that fluency. In our design, judges handle only the claims the lexicon cannot resolve, and their agreement with human labels is reported. A judge-only variant is run as a comparison arm: it is scored by its correlation with human critique-consistency labels on the audited subset.

Why not just use AesCanvas's evidence-grounded accuracy? It is the most serious alternative and measures something close. Its limit is scale: 301 expert cases, each needing curation, which is too few to stratify by attribute or genre. It also tests whether a cue is named, not whether the output changes when the cue changes. Our interventions are cheap to generate once validated. The grounded-critique baseline tells us whether the two measures agree. If they agree strongly, our contribution becomes scale and automation rather than a new property.

Why not just use saliency or attention maps? These apply poorly to written critiques and scalar attribute heads, and they show where a model looks, not whether its stated reason changes the score. Score mediation tests the latter directly.

Proposed Experiments and Decisive Checks

Experiment 1: protocol variance. For every model, we report the SRCC spread across resize policy, split, prompt paraphrase, and score-extraction rule on AVA, AADB and PARA. It tests one question: do published between-model gaps of a few percent survive this spread?

Experiment 2: faithfulness versus correlation. We compute the three faithfulness measures on the validated edit set, separately for each low-level attribute and for composition. We compare the resulting model ranking to the correlation ranking. Our hypotheses are:

  • models trained with reasoning chains will show high critique fluency but low consistency on composition recrops;
  • AGREE's per-attribute adapters will show higher attribute-direction consistency on its five attributes than end-to-end fine-tuned language models, but no advantage on composition.

Experiment 3: prediction of out-of-distribution drops. This is the decisive test of the central claim. We train on AVA and test on PARA and AADB, then ask whether faithfulness or in-distribution SRCC better predicts each model's drop. The core claim fails if either condition holds:

  • intervention consistency tracks SRCC so closely that it adds nothing;
  • human raters themselves show near-chance agreement on edit direction for composition recrops, which would make composition faithfulness ill-defined.

In either case, the scope narrows to the low-level attributes where ground truth is stable.

The edited images could be off-distribution in ways models react to artifactually. As a control, we include null edits: re-encoding and resizing that humans judge unchanged. Any model whose attributes move on null edits is flagged.

All code, prompts, edits and human judgments will be released with the harness.