Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment

Qiuyu Chen, Wei Zhang, Ning Zhou, Peng Lei, Yi Xu, Yu Zheng, Jianping Fan

Abstract

To leverage deep learning for image aesthetics assessment (IAA), a critical challenge is incorporating image aspect ratios to learn robust models. This paper introduces Adaptive Fractional Dilated Convolution (AFDC), an aspect-ratio-embedded, composition-preserving, and parameter-free approach that addresses this issue at the convolutional kernel level. By adaptively constructing fractional dilated kernels based on image aspect ratios, the model maintains the integrity of the original composition even when images are warped for batch processing. Experimental results on the AVA dataset demonstrate that AFDC achieves state-of-the-art performance.

The Challenge: Composition and Aspect Ratio

In standard computer vision tasks like object recognition, images are often warped or cropped to fit a fixed square input (e.g., 224x224). While a warped cat is still recognizable as a cat, in aesthetics assessment, composition is paramount.

The impact of standard data augmentation on aesthetic scores.

Distorting a wide landscape into a square alters the "rule of thirds," balance, and lead-in lines. If a high aesthetic score is assigned to a distorted version of a beautiful image, it introduces "label noise," which hinders the model's ability to learn true aesthetic principles. Previous attempts to solve this—such as processing images individually or using patches—either suffered from extreme computational inefficiency or lost global context.

Methodology: Adaptive Fractional Dilated Convolution (AFDC)

The core innovation is the AFDC module, which adapts the network to the image rather than forcing the image to fit the network.

  1. Fractional Dilation via Interpolation: Standard dilated convolutions use integer gaps between weights. AFDC uses fractional gaps to match the specific aspect ratio of the input. Since fractional pixels cannot be sampled directly, the model performs linear interpolation between the two nearest integer dilated kernels.
  2. Consistent Receptive Field: By adjusting dilation based on the original aspect ratio, the "effective receptive field" of the filter remains consistent with the original composition, effectively "undoing" the distortion caused by warping at the feature extraction stage.
Construction of the Adaptive Fractional Dilated Kernel via linear interpolation.
  1. Efficient Training: To maintain high throughput, the authors utilize a grouping strategy where images with similar aspect ratios are batched together. The kernel interpolation is implemented via a concise matrix formulation, making it compatible with standard deep learning libraries without adding new learnable parameters.
  2. Composition-Aware Modules: The architecture concludes with Spatial Pyramid Pooling (SPP) to capture aesthetic features across multiple scales, from global balance to local details.

Experimental Results

The model was evaluated on the AVA (Aesthetics Visual Analysis) dataset, containing approximately 250,000 images with human-provided ratings.

  • State-of-the-Art Performance: The AFDC-based model achieved a classification accuracy of 83.24%, surpassing previous benchmarks like NIMA and MNA-CNN.
  • Correlation Metrics: The model showed significantly improved Spearman’s Rank Correlation Coefficient (SRCC) and Linear Correlation Coefficient (LCC), indicating that its predictions align more closely with human aesthetic judgment.
  • Plug-and-Play Utility: Because the module is parameter-free, it can be easily integrated into popular backbones like ResNet or VGG to enhance their aesthetic perception capabilities.

Conclusion

AFDC provides a native, kernel-level solution to the problem of aspect ratio distortion in image aesthetics assessment. By preserving composition through adaptive dilation, the model eliminates label noise and achieves superior accuracy. Future work may extend this approach to related fields such as automatic photo enhancement and intelligent cropping, where spatial relationships are equally critical.

Summary updated

Abstract

To leverage deep learning for image aesthetics assessment, one critical but unsolved issue is how to seamlessly incorporate the information of image aspect ratios to learn more robust models. In this paper, an adaptive fractional dilated convolution (AFDC), which is aspect-ratio-embedded, composition-preserving and parameter-free, is developed to tackle this issue natively in convolutional kernel level. Specifically, the fractional dilated kernel is adaptively constructed according to the image aspect ratios, where the interpolation of nearest two integer dilated kernels are used to cope with the misalignment of fractional sampling. Moreover, we provide a concise formulation for mini-batch training and utilize a grouping strategy to reduce computational overhead. As a result, it can be easily implemented by common deep learning libraries and plugged into popular CNN architectures in a computation-efficient manner. Our experimental results demonstrate that our proposed method achieves state-of-the-art performance on image aesthetics assessment over the AVA dataset.
BibTeX
@inproceedings{chen2020adaptive,
  author = {Qiuyu Chen and Wei Zhang and Ning Zhou and Peng Lei and Yi Xu and Yu Zheng and Jianping Fan},
  title = {{Adaptive Fractional Dilated Convolution Network for Image Aesthetics Assessment}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=0oxxqkXQJVd},
}
Figure