Machine models that score how beautiful an image is are now used as automatic judges: they rank photographs, filter training data, and act as reward signals for image generators. They are validated almost entirely one way, by correlating their scores with average human ratings on fixed benchmark photos. That test says nothing about how a scorer reacts when an image is slightly damaged. A generator optimizing against the scorer will find exactly those reactions. This proposal imports a concept from structural engineering to measure it. Engineers know that thin shells and domes collapse far below their theoretical strength because of small dents and thickness variations. They summarize this with a "knockdown" curve: remaining capacity as a function of imperfection size. We propose to measure knockdown curves for aesthetic scorers and for people on the same imperfect images, and to treat the gap between the two curves as the missing validation.
The core claim is specific and falsifiable. Current aesthetic scorers are imperfection-sensitive in the wrong places. Their scores should change sharply under some small, localized geometric deformations that people barely notice, and stay flat under others that people clearly dislike. We further claim that the global photometric corruptions used in prior sensitivity analyses (brightness, blur, hue) are the wrong probe for finding this mismatch. Structural mechanics learned the same lesson: localized dents, not the smooth global buckling shape, govern real collapse. If both claims hold, benchmark correlation is an inadequate certificate for any scorer used as an optimization target. The fix is a cheap, human-anchored sensitivity test.
What Exists and What Is Missing
On the aesthetics side, models have improved steadily on consensus benchmarks. ILGNet in Deep Image Aesthetics Classification using Inception Modules and Fine-tuning Connected Layer fuses early local and deep global features. Image Aesthetic Assessment Assisted by Attributes through Adversarial Learning aligns predicted score–attribute distributions with human ones through a discriminator. Aes-R1 in Unlocking the Essence of Beauty trains a multimodal language model to reason about aesthetics and corrects its tendency to rate mediocre images highly. We reuse all three as subjects of study, not as competitors.
The closest prior work is AGREE, from When Attributes Disagree: Gradient Conflict in Image Aesthetic Assessment. It perturbs five global photometric attributes and uses the model's response to route features into separate adapters. It motivates this proposal, because it shows perturbation response carries real signal. It differs in two ways. It measures sensitivity only to global pixel-level transformations, and it never checks whether the model's sensitivity matches human sensitivity; the research file flags exactly this as open. Finding Badly Drawn Bunnies comes closest to linking geometric deviation with quality. In its GACL model, the length of a sketch's feature vector serves as a quality score and matches human rankings 92.61% of the time. But it assumes quality falls monotonically with distance from a class prototype. Beyond Geometry: Artistic Disparity Synthesis directly contradicts that assumption: professionals deliberately introduce geometric "errors" for effect. So the direction of the human response to an imperfection is an empirical question, and we measure it rather than assume it.
The method comes from structural mechanics. A posteriori initial imperfection identification in shell buckling problems represents dents as Gaussian radial basis functions, each a bump defined by a center, a width and an amplitude. It shows that seeding models with the first global buckling shape is "neither robust nor reliably accurate," because localized dents govern collapse. It also finds the error surface too bumpy for gradient search and uses a genetic algorithm instead. Bayesian Buckling Load Optimisation for Structures With Geometric Uncertainties supplies two more tools. Quasi-random Sobol sampling, which spreads samples evenly, estimates the mean buckling load within 0.3% from 128 samples. Gaussian-process Bayesian optimization then searches for designs efficiently.
Both mechanics papers also warn us: their own validation is purely simulated. This is the empirical validation gap described in The Empirical Validation Gap, reproduced inside papers that set out to close it. We avoid repeating it by making human judgments, not model outputs, the ground truth for every sensitivity claim.
Research Program
We define an imperfection as a displacement field applied to an image. It is a sum of Gaussian radial basis bumps, each warping pixel positions locally around its center. Its size $\epsilon$ is the peak displacement in pixels. Photometric control imperfections use the same bump shapes but modulate brightness or saturation instead of position. For a scorer $s$ and a source image $x$, the model knockdown at size $\epsilon$ is the expected fractional score loss under random imperfections of that size:
$$\kappa_s(\epsilon) = \mathbb{E}_{\delta \sim \mathcal{D}_\epsilon}\left[\frac{s(x) - s(x \oplus \delta)}{s(x)}\right].$$
Here $\mathcal{D}_\epsilon$ is the distribution of bump layouts at size $\epsilon$, and $x \oplus \delta$ is the warped image. The human knockdown $\kappa_h(\epsilon)$ is estimated from pairwise judgments: raters see the original and the perturbed image, choose the more pleasing, and we fit a Bradley–Terry model, which converts win rates into latent quality differences. Two quantities summarize the mismatch. The first is the area between the model and human knockdown curves. The second is the inversion rate: the fraction of perturbation pairs that the model and humans order oppositely.
We build three components. The first is a perturbation library covering localized geometric warps, localized photometric bumps, and AGREE's five global photometric edits. The second is a Sobol-based estimator of each curve's mean and spread. The third is a Bayesian-optimization search for the worst-case divergent imperfection: the bump layout of fixed size that maximizes model score change while a small human-validated set shows negligible human change, or the reverse. Human validation of the discovered worst cases is mandatory; nothing in the evaluation relies on a model judging a model. As a downstream test, we fine-tune one scorer on a small set of human-labeled perturbation pairs. This checks whether the curves can be brought together without losing benchmark correlation.
To keep "small" honest, a pilot human study first fixes, for each imperfection family, the size at which half of raters notice any change. All cross-family comparisons are then made at matched noticeability, not matched pixel displacement.
Baselines
The naive reference is the current practice: report Spearman and Pearson correlation on held-out benchmark images and stop. Reported figures include Spearman 0.7041 on AADB and 0.6313 on the AVA attribute subset for the adversarial model in Image Aesthetic Assessment Assisted by Attributes through Adversarial Learning. The Swin backbone on PARA's generic task reaches SROCC 0.9021 in Personalized Image Aesthetics Assessment with Rich Attributes. This baseline is beaten if two scorers with near-identical benchmark correlation show clearly different inversion rates on our human-validated perturbations. That would show benchmark correlation does not certify robustness.
The strongest existing scorers serve as subjects. We use AGREE, which reports 1.6% to 4.8% relative correlation gains and up to 36.5% lower error on hard samples (When Attributes Disagree), and Aes-R1, which reports average gains of 47.9% in Pearson and 34.8% in Spearman correlation over its Qwen2.5-VL-7B backbone (Unlocking the Essence of Beauty). We also use a plain monolithic regressor and ILGNet. For the sketch domain we use GACL on QuickDraw. All are run from released or retrained checkpoints on identical perturbed images. No reported result was found on Lacuna for knockdown curves or human–model perturbation agreement for any of these models, so our measurements are the first numbers of this kind.
The strongest competitor for the probe is AGREE-style global photometric perturbation, evaluated with the same Sobol budget and matched noticeability. The localized geometric family beats it if it reveals a significantly larger area between curves, or a higher inversion rate, at matched noticeability on at least two datasets. For worst-case search, the baselines are plain random sampling, Sobol sampling, and gradient search over the bump parameters using the scorer's gradients, each given the same number of scorer evaluations. The central ablation keeps the bump layout and search fixed and swaps geometric warps for localized photometric bumps. This separates "localized" from "geometric" as the source of any effect.
Why not just…?
Why not just run a standard adversarial attack on the scorer? Gradient attacks will find tiny pixel patterns that move scores. That is already expected, and it says nothing about whether people agree. Our question concerns perceptible, structured imperfections for which a human response exists and can be measured. That said, gradient search over the bump parameters is a real threat to the Bayesian-optimization component, which is why it is a baseline. The shell-buckling evidence (A posteriori initial imperfection identification) suggests the landscape is bumpy enough to defeat gradients, but images are not shells. If gradients match Bayesian optimization at equal budget, we will drop Bayesian optimization and report that.
Why not just use global photometric corruptions, as AGREE does? They are cheap and already implemented. Our hypothesis is that they play the role of the global buckling shape: easy to apply but not where the dangerous mismatch lies. The matched-noticeability comparison settles this directly. If global corruptions expose as much human–model divergence as localized warps, the geometric part of the proposal is unnecessary, and the contribution shrinks to the human-anchored knockdown protocol alone.
Why not just study the hard, ambiguous middle-score images already in the benchmarks? The survey in Image Aesthetic Assessment: An experimental survey shows that models struggle on images near 5/10. That failure is about level, not local slope. A scorer can rank the middle correctly and still respond wildly to small edits, and an optimizer exploits exactly that slope. We will check whether per-image inversion rate is predicted by the image's benchmark difficulty. If it is fully predicted, the hard-sample analysis already captures our signal.
Why not just collect human pairs on perturbed images and fine-tune, skipping the measurement? That is the eventual remedy, and our final experiment tests it. Without the curves, though, one cannot tell which perturbations to label, or whether fine-tuning fixed the mismatch or only moved it to other imperfection families. Held-out families show which.
Proposed Experiments
Source images come from AVA (AVA), AADB, PARA, and LAPIS paintings, with QuickDraw sketches as the geometric-quality domain. We draw roughly 400 source images stratified by benchmark score. Each receives imperfections at four noticeability-matched sizes in each family, and pairwise human judgments are collected with enough raters per pair to estimate win rates. PARA's rater profiles let us test whether human knockdown itself varies with personality, as PARA's findings on polarized ratings would predict. LAPIS and the stereographic evidence let us look for imperfections that people prefer. Such cases would falsify prototype-based quality models like GACL outright.
The decisive checks are four. First, we test whether model and human knockdown curves diverge in a way that benchmark correlation fails to rank. Second, we test whether localized geometric warps expose more divergence than global and localized-photometric edits at matched noticeability. Third, we test whether guided search finds human-validated worst cases in fewer scorer evaluations than random or Sobol sampling. For reference, the only budget figure on file is 128 Sobol samples for 0.3% mean error in the buckling setting. Fourth, we test whether fine-tuning on a few hundred human-labeled pairs shrinks the inversion rate on held-out imperfection families while keeping benchmark Spearman correlation within a small tolerance of the original. A negative result on the first check, with curves that track humans closely, would itself be useful: it would license current scorers as reward models against this class of imperfection.
References
- Deep image aesthetics classification using inception modules and fine-tuning connected layer
- Image Aesthetic Assessment Assisted by Attributes through Adversarial Learning
- Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
- When Attributes Disagree: Gradient Conflict in Image Aesthetic Assessment
- Finding Badly Drawn Bunnies
- Beyond Geometry: Artistic Disparity Synthesis for Immersive 2D-to-3D
- A posteriori initial imperfection identification in shell buckling problems
- Bayesian buckling load optimisation for structures with geometric uncertainties
- Personalized Image Aesthetics Assessment with Rich Attributes
- Image Aesthetic Assessment: An experimental survey
- AVA: A large-scale database for aesthetic visual analysis