Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment

Daniel Vera Nieto, Luigi Celona, Clara Fernandez-LabradorNeurIPS 2022

The Shift from Numbers to Words

Researchers have introduced the Reddit Photo Critique Dataset (RPCD), a massive collection of 74K high-resolution images paired with 220K professional and hobbyist critiques. By applying sentiment analysis to these text-based critiques, the authors demonstrate that language is a powerful, interpretable proxy for traditional numerical aesthetic scores. This approach bridges the gap between knowing how good a photo is and understanding why it is perceived that way.

The Problem with Traditional Aesthetic Assessment

Automated Image Aesthetic Assessment (IAA) has traditionally treated beauty as a regression problem: given an image, predict a single number (e.g., 1 to 10). However, aesthetics are deeply subjective. A single number fails to capture why a photo works (e.g., "great use of the rule of thirds") or why it fails (e.g., "distracting background").

Existing datasets often rely on simple ratings that are prone to bias and lack interpretability. Furthermore, humans are naturally better at expressing nuanced opinions through language than through arbitrary scales. The authors aim to move the field toward "explainable" aesthetics by leveraging the rich, constructive feedback found in online photography communities.

Examples of images, text critiques, and derived sentiment scores from the RPCD dataset.

Methodology: Sentiment as an Aesthetic Proxy

The core innovation of this research lies in treating sentiment polarity as a proxy for aesthetic quality.

  1. Data Acquisition: The authors built an automatic pipeline to scrape the /r/photocritique subreddit, filtering for top-level comments to ensure they captured actual critiques rather than conversational noise.
  2. Sentiment-to-Score Mapping: They used TwitterRoBERTa, a transformer model pre-trained on social media text, to analyze the sentiment of each critique. They mapped the probabilities of negative, neutral, and positive sentiments into a continuous "sentiment score" between 0 and 1.
  3. Validation: To prove these sentiment scores are meaningful, they applied the same technique to older datasets (AVA and PCCD) that already had human-assigned numerical ratings. They found a strong positive correlation, confirming that the "vibe" of a critique accurately reflects the numerical quality of the image.
  4. Modeling: They evaluated several architectures, including Vision Transformers (ViT), on two tasks: predicting the sentiment score from an image and generating aesthetic critiques (Aesthetic Image Captioning).

The Datasets

  • RPCD (Reddit Photo Critique Dataset): The new benchmark containing 73,965 images and 219,790 comments. Unlike previous datasets, these images are mostly UltraHD/FullHD, and the comments are significantly longer and more informative (averaging ~49 words).
  • AVA (Aesthetic Visual Analysis): A classic benchmark used for comparison. While huge, its comments are often short and its scores are heavily skewed toward the "average."
  • PCCD (Photo Critique Captioning Dataset): A smaller dataset of ~4.2K images used to validate the sentiment-scoring approach.

Key Insights and Results

The most significant takeaway is that language acts as a highly effective label. We don't necessarily need humans to provide 1-10 ratings if we have access to their written critiques; sentiment analysis can distill these critiques into usable training labels.

Furthermore, the authors found that RPCD has a much more balanced distribution of scores than older datasets. On sites like DPChallenge (the source of AVA), users mostly comment when they like a photo. On Reddit's /r/photocritique, the explicit goal is to find flaws, leading to a richer representation of "low-quality" or "mediocre" aesthetics that models usually struggle to learn.

Performance Highlights: * Strong Correlation: The sentiment-derived scores showed a Spearman’s Rank Correlation Coefficient (SRCC) of 0.6418 with human ratings on the AVA dataset, proving the method's validity. * ViT Superiority: The AestheticViT (a Vision Transformer Large) outperformed previous state-of-the-art models by roughly 4% in correlation metrics on the AVA dataset. * RPCD Complexity: On the new RPCD dataset, the best model achieved an SRCC of 0.250 for predicting sentiment scores. While this seems lower than older datasets, it reflects the much higher complexity and diversity of the RPCD critiques. * Captioning Challenges: In the critique generation task, models achieved a BLEU-1 score of 0.211. This suggests that while models can pick up on aesthetic keywords, generating human-level constructive criticism remains a difficult open challenge.

Limitations and Future Work

While promising, the approach has limitations. The dataset is primarily sourced from Reddit, which has a demographic skew toward Western, English-speaking users, meaning the captured "aesthetics" may not reflect global standards. Additionally, automated sentiment analysis isn't perfect—sarcasm or highly technical photography jargon might be misclassified by a general-purpose model like RoBERTa.

Future work could involve weighting critiques based on the "reputation" or "expertise" of the commenter, or moving beyond a single sentiment score to multi-dimensional scores that specifically rank style, color, and composition separately.

Summary updated

BibTeX
@inproceedings{nieto2022understanding,
  author = {Daniel Vera Nieto and Luigi Celona and Clara Fernandez-Labrador},
  title = {{Understanding Aesthetics with Language: A Photo Critique Dataset for Aesthetic Assessment}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure