Correct for Whom? Subjectivity and the Evaluation of Personalized Image Aesthetics Assessment Models

Samuel Goree, Weslie Khoo, David J. CrandallAAAI 2023 · DOI

TL;DR

This paper argues that current methods for evaluating personalized image aesthetic quality assessment (IAQA) models are flawed because they rely on benchmark datasets that don't adequately capture individual subjective preferences. They introduce a new dataset and analysis showing that existing "ground truth" labels are poor predictors of individual taste, and that personalization models, while improving accuracy for some, also increase variance and don't fully explain disagreements.

What Problem Does This Paper Address?

The field of Image Aesthetic Quality Assessment (IAQA) aims to automatically predict how visually appealing an image is. Traditionally, this involved predicting an average rating from many human observers, assuming a universal standard of beauty. More recently, research has shifted towards "personalized IAQA," acknowledging that aesthetic preferences are subjective and vary between individuals. These personalized models often use techniques like few-shot learning to adapt to a specific user's taste.

However, a critical question arises: how do we evaluate these personalized models? The authors argue that current evaluation practices, which often rely on existing benchmark datasets (like AADB), are problematic. They draw on philosophical critiques, particularly from feminist aesthetics, which highlight how seemingly objective standards can inadvertently encode dominant social values and biases. This leads to the core problem: do current benchmark datasets truly represent diverse subjective preferences, or do they simply reflect a specific, potentially biased, consensus? And if they don't, how can we reliably evaluate models that claim to personalize for individual taste?

Methods

The authors introduce a new dataset, PR-AADB, which is a relabeling of a subset of the existing AADB dataset. Key methodological differences include:

  • Pairwise Comparisons: Instead of asking users to rate images on a scale, participants were presented with pairs of images and asked to choose which they "enjoyed more," or if they enjoyed both, neither, or found them too different. This approach is inspired by methods in image quality assessment and aims to provide a more grounded comparison.
  • "Enjoyment" Prompt: The prompt focused on personal "enjoyment" rather than abstract "aesthetic quality" to better capture individual sensory experience, drawing parallels to John Dewey's concept of aesthetic experience.
  • Common and Unique Pairs: Each participant labeled a set of "common" image pairs (shared across all participants) for controlled training/personalization, and a larger set of "unique" image pairs (specific to that participant) for evaluation.
  • Demographic and Qualitative Data: Participants provided demographic information and answered free-text questions about their labeling process, allowing for deeper analysis of disagreements.
  • Inferred Scores: They developed a method to infer single-image scores from pairwise preferences using principal eigenvector analysis, allowing comparison with existing single-image scores.
  • Statistical Analysis of Disagreements: Logistic regression was used to explore whether demographic factors, aesthetic attributes of images, or image content could predict disagreements between the AADB ground truth and participant preferences.

Datasets

  • AADB (Aesthetic Attribute Database): Used as the source for images. The original dataset contains overall aesthetic ratings and 11 aesthetic attributes for images.
  • PR-AADB (Personalized Relabeled AADB): A newly introduced dataset. It consists of labels for 16,548 image pairs drawn from 8,835 images of the original AADB dataset. It includes pairwise preference labels, common and unique image pairs, and participant demographic data.

Key Insights and Ideas

  • "Ground Truth" is Not Universal: The core finding is that existing aggregate aesthetic scores (like those in AADB) are poor predictors of individual preferences. This challenges the notion of a single, objective aesthetic quality.
  • Personalization Amplifies Variance: While few-shot personalization can improve accuracy for some users, it also significantly increases the variance in performance across users. This suggests that personalization isn't a simple fix for subjective differences.
  • Disagreements are Hard to Predict: Demographic factors (age, gender, race, education) did not significantly predict whether a participant's judgment would align with the AADB ground truth. This implies that subjective differences are not easily explained by easily measurable user characteristics.
  • Subtle Factors Matter: While broad demographic categories don't explain disagreements, specific aesthetic attributes (like good content, color harmony, motion blur) and image content (e.g., nature vs. military themes) do have some predictive power, though the overall explanatory power is low.
  • The "Computational Construction of Taste": The authors warn that IAQA datasets and models might inadvertently "construct" taste by privileging certain visual patterns or interpretations, rather than objectively measuring existing preferences. This echoes concerns about how benchmark datasets shape research directions.

Key Results

  • Low Correlation Between Scores: The correlation between AADB single-image scores and scores inferred from the authors' pairwise labels was only 0.27.
  • Variable Accuracy for AADB Scores: AADB scores produced accuracy values ranging from 0.2875 to 0.575 for predicting pairwise preferences, a difference of almost 30%, with some participants performing worse than random guessing.
  • Personalization Increases Variance: When a personalized IAQA model was fine-tuned on common image pairs, the accuracy on unique image pairs did not significantly improve, but the variance in accuracy across participants increased from a standard deviation of 0.065 to 0.129.
  • Low Explanatory Power for Disagreements: A logistic regression model attempting to predict disagreements between AADB ground truth and participant preferences had a pseudo-R² of only 0.023, indicating that demographic, aesthetic, and content features explained very little of the variance in consistency.

Limitations and Future Work

  • Sample Representativeness: The authors acknowledge that their participant sample, drawn from a university mailing list, may not be representative of the general population, limiting strong claims about demographic factors.
  • Prompt Interpretation: The free-text responses revealed that participants interpreted the "enjoyment" prompt in diverse ways, highlighting the challenge of controlling for subjective interpretation even with carefully chosen wording.
  • Future Work: The authors encourage future work to:
  • Investigate IAQA algorithms through user studies with specific populations rather than aiming for general models.
  • Explore the potential downstream social consequences of predicting aesthetic preferences, particularly in social media contexts.
  • Further study the difficulty of accounting for subjectivity in machine learning, as current approaches may enforce reductive models of subjective difference.

Summary updated

BibTeX
@inproceedings{goree2023correct,
  author = {Samuel Goree and Weslie Khoo and David J. Crandall},
  title = {{Correct for Whom? Subjectivity and the Evaluation of Personalized Image Aesthetics Assessment Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i10.26395},
}
Figure