Content-Gated Pyramid Pooling for Aesthetic Assessment of Unwarped Photographs

Models that score how good a photograph looks almost always start by squashing the photo into a square. That damages the very things aesthetics depends on: a panorama's horizon gets compressed, a tall portrait's subject gets stretched, and the model never learns what shape the frame was. Earlier work shows the damage is real and grows with how far a photo is from square. What it does not show is why keeping the native shape helps. One possibility is that the pixels are no longer distorted. Another is that the model can now tell what shape the frame is. A third is that it can see where things sit relative to the frame. The answer also likely depends on what the photo shows: a wide frame is a virtue for a landscape and a liability for a snapshot.

We propose a single-pass aesthetic scorer that runs a standard backbone over the unwarped image. It then summarizes the feature map with spatial pyramid pooling, an old trick that cuts any-sized feature map into fixed grids of cells (whole image, thirds, quarters) and pools each cell, so the output has a fixed length while keeping coarse layout. In our version, a compact description of the photo's content decides which cells matter. Around this model we build a controlled study that separates the three sources of benefit. The deliverable is an architecture and an answer to whether "native aspect ratio" gains are about pixels, frame shape, or content-dependent layout.

Core Claim

Our claim is that most of the benefit of preserving aspect ratio in aesthetic assessment comes from content-dependent layout. Concretely, the payoff comes from knowing where things sit relative to the frame, and from weighting those positions differently for different kinds of content. Undistorted pixels alone account for less. If this is right, three results should follow. First, a pyramid head whose cell weights are set by the photo's content should beat both a native-shape model with plain global pooling and a squashed model that is simply told the original aspect ratio. Second, the margin should be largest on extreme aspect ratios and on content categories with strong framing conventions. Third, the learned cell weights should differ in interpretable ways across categories.

If instead the squashed-plus-aspect-ratio model matches the native-shape models, then the field's native-resolution machinery is mostly buying metadata. That would be a useful negative result.

Where This Sits

Three lines of work motivate the proposal.

The first establishes the cost of squashing. Aspect-Ratio-Preserving Multi-Patch Image Aesthetics Score Prediction samples unwarped square patches and finds that the error reduction on extreme-ratio images is up to 7.9 times larger than on ordinary 3:2 or 4:3 photos. MPA-Net extends this approach but runs into two problems. Training patches from the same image together caused features to cancel. It also had to add a squashed global view at test time to recover overall composition, which reintroduces the distortion it set out to remove. Charm brings native-shape processing to vision transformers. It keeps focal regions at full resolution, downsamples the background, and adds scale and position embeddings. Charm is the strongest current competitor, but it chooses which regions to keep largely at random and has no notion of content category. None of these three asks which part of their gain comes from frame-shape knowledge rather than from undistorted pixels.

The second line connects content to aesthetic criteria. Deep Aesthetic Quality Assessment With Semantic Information trains aesthetic and category prediction jointly and learns that categories such as "Landscape" and "Snapshot" shift scores in opposite directions. However, it squashes its inputs, and its semantics only touch the shared trunk, not where the model looks.

The third line uses pyramids for aesthetics. Modeling Image Composition for Visual Aesthetic Assessment uses atrous pyramid pooling (parallel dilated convolutions at several scales) and a graph over image regions, but on fixed-size inputs and without content conditioning. Its view that layout dominates aesthetics is in tension with evidence that a perfectly composed image can still fail when its narrative falls apart (She et al.). Our content gate is designed to test that tension: if layout weights do not need to vary with content, the gate should learn nothing and the ablation will show it.

We reuse methods from four sources. From SPP-net we take the core mechanism: convolution runs once over an arbitrary-shape image, and pyramid cells pool it into a fixed-length vector. From Spatial Moment Pooling we take a pooling refinement. That paper shows that adding variance and higher moments to global pooling raises NIMA's rank correlation from 0.864 to 0.890, but it pools over the whole map and discards layout. We pool moments inside each pyramid cell instead. From NIMA we take the training target: a 10-bin score distribution fit with an Earth Mover's Distance loss, which compares distributions while respecting the order of the bins. From NaViT we take the idea of grouping variable-shape images into the same batch. This addresses SPP-net's own warning that training through variable input sizes is awkward.

One paper constrains the design. AGREE shows that a single scalar aesthetic loss produces conflicting gradients between attributes, and it explicitly leaves composition and semantics unrouted. We therefore treat layout and content as a separate pathway rather than folding them into the backbone loss.

Research Program

Model. The model processes the image at native aspect ratio with its shorter side fixed (for example, 448 pixels) and the longer side free. We use two backbones. One is a convolutional network, matching the MPA-Net lineage. The other is DINOv2-large with interpolated position embeddings, matching Charm.

The final feature map is pooled over fixed grids defined in coordinates relative to the frame: 1×1, 3×3 (a rule-of-thirds grid), and 4×4. The output length is therefore fixed regardless of shape. Each cell contributes its feature mean and standard deviation. The log aspect ratio $r$ is passed to the head explicitly, because relative-coordinate cells on a 3:1 panorama are elongated and would otherwise hide the frame shape.

Content gate. A content vector $s$ comes from a frozen CLIP-style image embedding. When AVA category tags are available, we add an auxiliary prediction of those tags. The gate modulates each cell's pooled feature $z_{l,b}$ at pyramid level $l$ and cell $b$:

$$\tilde z_{l,b} = \gamma_{l,b}(s, r)\odot z_{l,b} + \beta_{l,b}(s, r),$$

Here $\gamma$ and $\beta$ are small per-cell scale and shift vectors produced by a two-layer network from $s$ and $r$. The concatenated $\tilde z$ feeds a head that predicts the 10-bin distribution.

Batching and cost. Images are grouped by aspect ratio so that each batch is a dense tensor. This avoids both MPA-Net's per-patch updates and the cost of running $N$ separate passes per image.

Analyses beyond accuracy. We produce three analyses alongside the scores. The first is the aspect-ratio decomposition described under Proposed Experiments. The second inspects the learned gates per AVA category, for example whether landscapes put weight on the upper and lower thirds rows while portraits put weight on central cells. The third is a frame-perturbation probe. It pads or symmetrically crops held-out images to change composition while keeping content, then checks whether the score changes in the direction that framing conventions predict. This probe has no human labels, so it is diagnostic only.

Baselines

Squashed baseline. The naive reference is the same backbone and head with global average pooling on images squashed to a square. This is the standard pipeline, and every gain is measured against it.

Squashed plus aspect ratio. We add a stronger naive reference: the squashed model with $r$ concatenated into the head. It is cheap, and it isolates frame-shape knowledge from undistorted pixels. Any method that cannot beat it has not shown that native processing matters beyond metadata.

Charm. The strongest existing method is Charm on DINOv2-large, which reports a Pearson correlation (PLCC) of 0.783 on AVA. We will rerun it with its released configuration on our AVA split, using the same backbone weights, image resolution cap, number of epochs, and hyperparameter search budget as our model (an equal number of trials over learning rate and weight decay). We will report our reproduced value next to the published 0.783. Beating Charm means a higher PLCC and rank correlation (SRCC) than the reproduced Charm on full AVA, averaged over three seeds with non-overlapping bootstrap intervals, plus a clear margin on the extreme-ratio subset.

Convolutional comparisons. On the convolutional side, we compare to MPA-Net's multi-patch inference, which reports an LCC of 0.7096 and SRCC of 0.7004 on AVA (MPA-Net). We reimplement it on the same convolutional backbone so that the patch-versus-pyramid comparison is not confounded by architecture.

Accuracy-based references. For papers that report classification accuracy, we quote their figures as context only, since their protocols differ: ASPP FCN-GC at 83.59% (Liu et al.) and MTRLCNN at 79.08% (Kao et al.).

Ablations. These isolate the key ingredient. We compare native shape with global pooling against native shape with the ungated pyramid, and against the gated pyramid. We also replace gating with simple concatenation of $s$, drop the moment statistics, and drop $r$.

Why not just…?

Why not just squash the image and feed the aspect ratio as a number? This is the most serious threat. Nothing in the file rules it out, because no cited paper separates frame-shape knowledge from pixel distortion. It is a full baseline here. If it matches native processing, our core claim is false. We expect it to trail on the extreme-ratio subset, where squashing distorts objects most. The 7.9× disparity reported by Wang et al. suggests pixels matter there, but that study did not include this control.

Why not just run a native-shape backbone with global pooling, or use Charm as is? Global pooling preserves pixels but collapses layout. Spatial Moment Pooling shows that even within-map variance carries signal that mean pooling discards. Charm keeps layout implicitly through its tokens but has no content conditioning. The ablation that separates native global pooling, the ungated pyramid, and the gated pyramid answers whether explicit layout pooling adds anything once pixels are native. If the ungated pyramid ties Charm, we will report the pyramid as only a cheaper equivalent.

Why not just concatenate a semantic embedding to the features instead of gating cells? This is a real alternative, and it is close in spirit to Kao et al.. Gating should win only if the importance of layout truly varies with content. We test this directly on per-category subsets of AVA and on the theme-structured TAD66K. ELTA is reported to lag the EAT model on TAD66K because it lacks theme awareness, which suggests the benchmark rewards content-conditioned behavior. If concatenation ties gating, we drop the gate.

Why not just ask a multimodal language model to score the image? ArtQuant shows that structured description training helps such models on artistic benchmarks. However, the file contains no reported result for a multimodal model scoring native-aspect photographs on AVA, so we cannot dismiss it on evidence. We include a frozen-CLIP linear probe on native images as a cheap proxy and treat full multimodal fine-tuning as out of scope.

Proposed Experiments

Datasets. Training and main evaluation use AVA (Murray et al.), whose category tags provide content labels for the auxiliary loss and for subset analysis. We stratify results into aspect-ratio buckets following Wang et al.: 0.4–0.6, 0.6–0.8, and 0.8–1.0, measured as short side over long side. We also evaluate on TAD66K for theme dependence (Charm reports a relative 14.8% accuracy gain there but no absolute number in the file) and on AADB (no reported result found on Lacuna for a comparable absolute AADB score). Metrics are SRCC, PLCC, EMD, and binary accuracy at the score-5 threshold. We report the threshold accuracy because both multi-patch papers note instability at that boundary. Following AGREE, we also report results on a hard-sample subset of images whose ratings have high variance.

Decomposition test. The decisive check is the aspect-ratio decomposition. We compute the fraction of the gap between the squashed baseline and the gated native model that is closed by each intermediate model in turn: squashed plus $r$ (frame-shape knowledge), native with global pooling (pixel fidelity), and the ungated pyramid (layout). We compute these fractions separately per aspect bucket and per category.

Outcome criteria. The claim survives if three conditions hold. The gated model beats reproduced Charm and the squashed-plus-$r$ baseline. Layout and gating account for most of the gap on the extreme bucket. Gate patterns separate categories, as measured by linear decodability of category from $\gamma$ above chance on held-out images.

The claim fails cleanly in either of two cases. If squashed-plus-$r$ closes most of the gap, frame-shape knowledge rather than layout explains the native-ratio benefit. If gating adds nothing over concatenation, content does not change where the model should look. Either result would tell practitioners which part of native-resolution processing is worth its cost.