Abstract
To leverage deep learning for image aesthetics assessment (IAA), a critical challenge is incorporating image aspect ratios to learn robust models. This paper introduces Adaptive Fractional Dilated Convolution (AFDC), an aspect-ratio-embedded, composition-preserving, and parameter-free approach that addresses this issue at the convolutional kernel level. By adaptively constructing fractional dilated kernels based on image aspect ratios, the model maintains the integrity of the original composition even when images are warped for batch processing. Experimental results on the AVA dataset demonstrate that AFDC achieves state-of-the-art performance.
The Challenge: Composition and Aspect Ratio
In standard computer vision tasks like object recognition, images are often warped or cropped to fit a fixed square input (e.g., 224x224). While a warped cat is still recognizable as a cat, in aesthetics assessment, composition is paramount.
Distorting a wide landscape into a square alters the "rule of thirds," balance, and lead-in lines. If a high aesthetic score is assigned to a distorted version of a beautiful image, it introduces "label noise," which hinders the model's ability to learn true aesthetic principles. Previous attempts to solve this—such as processing images individually or using patches—either suffered from extreme computational inefficiency or lost global context.
Methodology: Adaptive Fractional Dilated Convolution (AFDC)
The core innovation is the AFDC module, which adapts the network to the image rather than forcing the image to fit the network.
- Fractional Dilation via Interpolation: Standard dilated convolutions use integer gaps between weights. AFDC uses fractional gaps to match the specific aspect ratio of the input. Since fractional pixels cannot be sampled directly, the model performs linear interpolation between the two nearest integer dilated kernels.
- Consistent Receptive Field: By adjusting dilation based on the original aspect ratio, the "effective receptive field" of the filter remains consistent with the original composition, effectively "undoing" the distortion caused by warping at the feature extraction stage.
- Efficient Training: To maintain high throughput, the authors utilize a grouping strategy where images with similar aspect ratios are batched together. The kernel interpolation is implemented via a concise matrix formulation, making it compatible with standard deep learning libraries without adding new learnable parameters.
- Composition-Aware Modules: The architecture concludes with Spatial Pyramid Pooling (SPP) to capture aesthetic features across multiple scales, from global balance to local details.
Experimental Results
The model was evaluated on the AVA (Aesthetics Visual Analysis) dataset, containing approximately 250,000 images with human-provided ratings.
- State-of-the-Art Performance: The AFDC-based model achieved a classification accuracy of 83.24%, surpassing previous benchmarks like NIMA and MNA-CNN.
- Correlation Metrics: The model showed significantly improved Spearman’s Rank Correlation Coefficient (SRCC) and Linear Correlation Coefficient (LCC), indicating that its predictions align more closely with human aesthetic judgment.
- Plug-and-Play Utility: Because the module is parameter-free, it can be easily integrated into popular backbones like ResNet or VGG to enhance their aesthetic perception capabilities.
Conclusion
AFDC provides a native, kernel-level solution to the problem of aspect ratio distortion in image aesthetics assessment. By preserving composition through adaptive dilation, the model eliminates label noise and achieves superior accuracy. Future work may extend this approach to related fields such as automatic photo enhancement and intelligent cropping, where spatial relationships are equally critical.
Summary updated