RAPID: Rating Pictorial Aesthetics using Deep Learning

Xin Lu, Zhe Lin, Hailin Jin, Jianchao Yang, James Z. WangACM Multimedia 2014 · DOI

TL;DR

This paper introduces RAPID, a deep learning system for automatically rating the aesthetic quality of images. It tackles the challenge of capturing both global composition and local details by employing a novel double-column convolutional neural network (CNN) architecture that processes these heterogeneous visual cues simultaneously. The system achieves state-of-the-art performance on the AVA dataset, demonstrating the power of deep learning for aesthetic assessment.

What Problem Does This Paper Address?

Assessing the aesthetic quality of an image is a complex task that has many practical applications, such as improving image retrieval systems or photo editing software. Traditional approaches relied on handcrafted features designed by experts, which often had limited scope and were difficult to implement computationally. While generic image features offered an improvement, they might not be optimal for the nuanced task of aesthetic evaluation. The authors aim to overcome these limitations by automatically learning effective aesthetic features directly from images using deep learning, addressing the need for a more principled and powerful approach to aesthetic quality rating.

Methods

The core of RAPID is a double-column deep convolutional neural network (DCNN). This architecture is designed to handle heterogeneous inputs representing different aspects of an image:

  • Global Views: These capture the overall composition and structure of an image. The authors experimented with three methods to generate global views: center-crop, warp, and padding.
  • Local Views: These focus on fine-grained details and textures within an image. These are generated by randomly cropping patches from the original high-resolution image.

The DCNN has two independent columns, each processing either the global or local views through their convolutional and initial fully-connected layers. The outputs from these columns are then combined and jointly trained in the final fully-connected layer. This allows the network to learn features that leverage both broad compositional elements and intricate local details.

Additionally, the authors explore incorporating style attributes (e.g., rule of thirds, motion blur) to further enhance aesthetic categorization. They propose a regularized DCNN (RDCNN) where a pre-trained style classifier's features are used to regularize the training of the aesthetic classifier. This approach is particularly useful given that style labels are only available for a subset of the dataset.

Datasets

  • AVA Dataset: This is the primary dataset used for training and evaluation. It's a large-scale dataset containing approximately 250,000 images with aesthetic ratings (ranging from 1 to 10). A subset of 14,000 images also includes style labels. The authors use a training set of 230,000 images and a testing set of 20,000 images, defining "low-quality" and "high-quality" images based on mean ratings. For style attribute learning, a subset of 11,000 images for training and 2,500 for testing are used.

Key Insights and Ideas

  • Global vs. Local Cues are Crucial: The paper strongly emphasizes that aesthetic quality is not determined by a single type of visual cue. Both the overall composition (global view) and the fine details (local view) are essential for accurate assessment.
  • Double-Column Architecture for Heterogeneity: The DCNN elegantly handles the need to process different types of visual information by dedicating separate processing pathways (columns) before integrating them. This is a key innovation for tasks involving multi-faceted visual understanding.
  • Data Augmentation via Random Cropping: Using randomly cropped local views (lr) as input proved to be an effective data augmentation strategy, significantly improving performance and helping to mitigate overfitting.
  • Style Attributes as Regularizers: Leveraging style attributes, even from a limited subset of data, can significantly boost the performance of aesthetic quality classification, demonstrating the interconnectedness of different visual qualities.
  • Learned Features are Smoother: The convolutional filters learned for aesthetic quality are observed to be smoother and less prone to radical intensity changes compared to those learned for object classification, suggesting that aesthetic differences lie more in texture and complexity than in distinct object boundaries.

Key Results

  • State-of-the-Art Performance: RAPID (specifically the RDCNN variant) achieved state-of-the-art accuracy on the AVA dataset for aesthetic quality categorization, outperforming previous methods.
  • DCNN Outperforms SCNN: The double-column DCNN consistently outperformed single-column CNNs (SCNNs) for both low and high quality image classification thresholds.
  • RDCNN Further Improves Accuracy: Incorporating style attributes via the RDCNN led to further improvements in aesthetic quality categorization accuracy, demonstrating the value of this regularization approach. For example, with $\delta=0$, DCNN achieved 73.25% accuracy, while RDCNN reached 74.46%.
  • Local View (lr) is Highly Effective: The random crop input (lr) yielded the highest accuracy among the tested input types for SCNN, highlighting its effectiveness as a data augmentation technique.

Limitations and Future Work

  • Computational Cost: Training deep neural networks is computationally intensive. The paper notes that training SCNN takes about two days, DCNN takes three days, and RDCNN (including style training) takes around three to four days.
  • Style Attribute Data Scarcity: While style attributes are beneficial, they are only available for a small fraction of the AVA dataset, necessitating techniques like inductive transfer learning.
  • Future Work: The authors suggest that the DCNN architecture can be easily extended to multi-column networks by incorporating more types of normalized inputs. They also mention that different architectures can be used in individual columns, allowing for more flexibility. The paper implicitly suggests further exploration into how different style attributes contribute to aesthetics and how to best leverage them.

Summary updated

BibTeX
@inproceedings{lu2014rapid,
  author = {Xin Lu and Zhe Lin and Hailin Jin and Jianchao Yang and James Z. Wang},
  title = {{RAPID: Rating Pictorial Aesthetics using Deep Learning}},
  booktitle = {ACM Multimedia},
  year = {2014},
  doi = {10.1145/2647868.2654927},
}

Loading related content...

Figure