Reading Detail in Context: Style-Conditioned Patch Aggregation for Aesthetic Judgment Beyond GPU Memory

When a person judges a high-resolution painting or photograph, they step back to see the whole picture and then lean in to inspect the brushwork or the sharpness. What they see up close means different things depending on what they saw from afar. A loose, visible stroke is a flaw in a photorealistic portrait and a strength in an impressionist landscape. Current automatic aesthetic models get only half of this. Style- and theme-aware models condition their judgment on the kind of image they are looking at, but they work on shrunken or single-pass inputs that throw away fine detail. Models from computational pathology can handle gigapixel images, meaning images far too large to fit in GPU memory, by cutting them into thousands of tiles and pooling the tile features. Those pooling rules, however, read every tile the same way no matter what the whole image is.

This note proposes that the whole-image view should decide how each high-resolution tile is interpreted, and not only which tiles get looked at. Concretely, a thumbnail pass produces a style-and-theme context vector. That vector rescales the features of every native-resolution tile before an attention-based pooling step, a form of multiple instance learning (learning from a label attached to a whole bag of unlabeled pieces). This setup lets the same local texture count for or against an image depending on its style. Before building anything, the note first asks whether fine detail changes human aesthetic judgment at all once the whole image is visible. It then asks whether context has to enter at the tile level, or whether simply attaching a style label at the end is enough.

Core Claim

The gap is specific. Context-dependent aesthetics shows that the same attribute can carry opposite value across categories: saturation helps landscapes and hurts portraits (Photo Aesthetics Ranking Network with Attributes and Content Adaptation), and viewers tolerate less stylization on faces than on cityscapes (Experiments in Objet Trouvé Browsing). Three lines of work motivate conditioning on context. Style-disentangling art models condition on medium: SAAN uses adaptive instance normalization (a layer that rescales features using statistics taken from a style input) (Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Method), and AANSPS uses category-specific attribute heads (Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories). Theme-aware full-resolution prediction feeds theme labels into a single forward pass over the whole photograph (Theme-Aware Aesthetic Distribution Prediction With Full-Resolution Photographs). All of these apply context to a single global representation of an image that has been downsampled or padded. None of them scales past GPU memory.

The pathology aggregators supply the method but not the conditioning. DSMIL pools thousands of tile embeddings using attention built from distances to the most critical tile (Dual-Stream Multiple Instance Learning Network for Whole Slide Image Classification With Self-Supervised Contrastive Learning). Hierarchical regional transformers group neighboring tiles into region embeddings (Towards Hierarchical Regional Transformer-based Multiple Instance Learning). PTree-Net comes closest in spirit: it uses a low-resolution thumbnail to produce a heatmap that decides where to zoom (Diagnose Like A Pathologist: Weakly-Supervised Pathologist-Tree Network for Slide-Level Immunohistochemical Scoring). In PTree-Net the thumbnail picks the location of the evidence. In our proposal it changes what the evidence means. A-Lamp, the one aesthetic model that looks at native-resolution patches, is limited to five patches chosen by a salient-object detector, and it scores them without style conditioning (A-Lamp: Adaptive Layout-Aware Multi-patch Deep Convolutional Neural Network for Photo Aesthetic Assessment).

Research Program

The model has three parts. First, a thumbnail encoder sees the whole image at low resolution. It outputs a context vector $c$ and is trained with an auxiliary style or theme loss wherever labels exist: BAID media, APDD's 24 categories, and AVA themes. Second, a frozen patch encoder turns $N$ native-resolution tiles into features $h_i$ plus normalized frame coordinates $p_i$. These features are cached to disk, which keeps GPU memory bounded by the aggregator rather than by image size. Third, a context-modulated attention aggregator combines the tiles:

$$ \tilde h_i = \gamma(c)\odot h_i + \beta(c) + e(p_i), \qquad z = \sum_{i=1}^{N} \frac{\exp\big(w^\top \tanh(V\tilde h_i)\big)}{\sum_j \exp\big(w^\top \tanh(V\tilde h_j)\big)}\,\tilde h_i . $$

Here $\gamma, \beta$ are small networks that map context to a per-channel scale and shift, $e(p_i)$ is a learned embedding of the tile's position within the frame, $V$ and $w$ are the attention parameters, and $z$ is the bag representation. The pooled vector $z$ is concatenated with the thumbnail features and fed to a head. That head predicts a rating distribution trained with Earth Mover's Distance on AVA, following the theme-aware work, or a scalar or attribute scores on BAID and APDD. The positional term addresses a known weakness of standard multiple instance learning, which treats tiles as an unordered set. Composition depends on where things sit relative to the frame. Because $c$ enters before attention, context can change both which tiles are selected and the direction in which each selected tile pushes the score. Late fusion cannot do the first of these.

The program has two stages. The first stage is a human study that decides whether the problem is real. We will collect a few hundred high-resolution artwork scans and photographs, which require new collection because the file lists no gigapixel aesthetic benchmark. Each image is rated twice by independent raters: once viewed as a screen-sized thumbnail, and once with free zoom. The difference between the two ratings, split by style, becomes the target that the second stage must explain. The second stage builds the model and tests it on existing benchmarks and on controlled probes.

Baselines

The naive reference point is the same thumbnail encoder and head with no tile pathway. It shares the training data and tuning budget with our model, so any gain can be attributed to native-resolution evidence. A second naive reference averages tile features with mean pooling, which is the failure mode the pathology literature warns about when informative regions are sparse.

The strongest existing aesthetic methods are the published context-aware models, each re-run on its own benchmark at its own input resolution. On AVA the theme-aware full-resolution model reports SRCC 0.7737, and 0.7469 without its theme module (Theme-Aware Aesthetic Distribution Prediction With Full-Resolution Photographs). On BAID, SAAN reports SRCC 0.473 and 76.80% accuracy (Towards Artistic Image Aesthetics Assessment: a Large-scale Dataset and a New Method). On APDD, AANSPS reports total-score SROCC 0.6085 (Paintings and Drawings Aesthetics Assessment with Rich Attributes for Various Artistic Categories). These are the numbers to beat. Beating them means exceeding each figure on its native benchmark under matched splits. We will also report our re-implementation's numbers, so that differences in the training pipeline do not masquerade as gains. On the zoom-rating set, beating them means explaining significantly more of the zoom-minus-thumbnail rating difference.

The strongest existing aggregator is DSMIL's dual-stream pooling, applied without conditioning to the same cached tile features. We add an attention pooling baseline alongside it. DSMIL reports 89.9% on Camelyon16 against 84.5% for attention pooling. That is a pathology result and does not transfer to aesthetics; no reported result exists for tile-based aggregators on aesthetic benchmarks. Two ablations isolate the key ingredient. Late fusion concatenates $c$ only with $z$ after context-blind attention. Localization-only conditioning follows PTree-Net: $c$ weights tile attention but does not rescale features. Every variant gets the same frozen encoder, the same tile budget, and the same number of hyperparameter trials.

Why not just…?

Why not just downsample, since raters judge holistically? This is the most serious threat, and the proposal is built around it. BAID's votes were cast on web-resolution images. For such labels, detail invisible on screen cannot matter except through correlated cues. The evidence that resolution helps is suggestive but confounded. The theme-aware model's gain over NIMA, 0.612 to 0.7737, bundles resolution with its shape and theme modules. The zoom-versus-thumbnail rating study settles this question directly. If the rating difference is negligible across styles, we report that as the main finding and the tile pathway is unnecessary. We also expect the tile pathway to add little on BAID and to show its value on the zoom-rated set and in the defect probes.

Why not just fine-tune SAAN or the theme-aware model at the largest resolution that fits in memory? Up to a few thousand pixels on a side, this may match our model, and we will run it as the high-memory baseline. Beyond that point, padding to a common tensor size fails for lack of memory, as the theme-aware paper's own design implies. We will plot accuracy against input resolution for both approaches to locate where they cross over.

Why not just run DSMIL unconditioned and append a style classifier at the end? This is the late-fusion ablation. An output layer could in principle learn style-by-evidence interactions. However, the attention step would still pick tiles without knowing the context. So when the same texture should be sought out in one style and ignored in another, late fusion should fail. The context-swap and defect probes below are designed to expose this. If late fusion matches our model on those probes, then early modulation is not needed and we would drop the claim.

Decisive Checks

We will run four checks. The defect-injection probe inserts small defects at native resolution into BAID and APDD images: local blur, compression blocks, and simulated retouching. The same textures, written as intentional marks, go into styles where they belong, for example stippling in sketches. We measure whether the predicted score drops for defects and not for intentional marks, and whether that sensitivity disappears at thumbnail resolution. The context-swap probe pairs a painting's tiles with a thumbnail from a different style. It checks whether the score shifts in the direction the category-specific rubrics predict, and whether tile-attention maps reorganize. Late fusion should show no reorganization of attention. The memory test measures peak GPU memory and latency as tile count grows from hundreds to tens of thousands. The fourth check is the zoom-rated set, where the question is whether per-style prediction of the rating difference is best for context-modulated aggregation. The proposal succeeds if all of the following hold: humans show a style-dependent zoom effect; our model tracks it; the late-fusion and localization-only ablations do not; and the thumbnail-only model's scores on AVA, BAID, and APDD are matched or exceeded.