AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception

Yipo Huang, Xiangfei Sheng, Zhichao Yang, Quan Yuan, Zhichao Duan, Pengfei Chen, Leida Li, Weisi Lin, Guangming Shi

Introduction

While Multimodal Large Language Models (MLLMs) like LLaVA or GPT-4V excel at object recognition, they often struggle with the abstract and subjective nature of Image Aesthetics Perception (IAP). Current models frequently fail to articulate the reasoning behind an image's beauty or how its composition influences emotional response. This gap stems from training data that prioritizes generic descriptive captions over nuanced aesthetic critiques. To address this, researchers have developed AesExpert, a foundation model designed to understand artistic intent, technical quality, and emotional resonance.

Methodology

The development of AesExpert involved a three-stage pipeline to cultivate "aesthetic expertise" in AI:

  1. Human Feedback Collection (AesFeedback): 48 participants, including art professionals, provided multi-layered feedback on 21,904 images. Moving beyond simple ratings, they addressed progressive questions ranging from coarse-grained aesthetic grades to fine-grained reasoning about clarity, color, and composition, and finally to emotional impact.
  2. Instruction Generation (AesMMIT): Using GPT-4, these human critiques were transformed into a massive dataset of 409,000 diverse instruction-response pairs (AesMMIT).
  3. Model Training: Several open-source MLLM backbones (LLaVA-1.5 and mPLUG-Owl2) were fine-tuned on this specialized dataset. By freezing vision encoders and optimizing the language model and projector, the models learned to translate visual features into sophisticated aesthetic discourse.
AesExpert performance overview and qualitative examples across natural, artistic, and AI-generated images.

Datasets and Benchmarking

The research introduces two primary datasets: AesMMIT, containing 409K instructions, and AesFeedback, the raw database of 88K natural language critiques. These cover a diverse array of Natural Images (NIs), Artistic Images (AIs), and AI-Generated Images (AGIs). Evaluation was conducted using AesBench, a benchmark specifically designed to test aesthetic perception, empathy, assessment, and interpretation.

Key Insights

  • The "How" Factor: Significant performance gains were observed in "How-style" questions, indicating that while baseline models identify content, they lack the vocabulary for aesthetic improvement or emotional explanation without specific tuning.
  • Human-Centric Labels: Human-generated critiques proved superior to machine-generated captions (like AVA-Comments) in capturing the "why" behind judgments and reducing model hallucinations.
  • AIGC Proficiency: AesExpert demonstrated a marked improvement in critiquing AI-generated images, successfully pinpointing subtle technical inconsistencies often missed by general models.
  • Compositional Focus: Statistical analysis identified "composition" as the most frequent term in human critiques, underscoring its role as a primary pillar of aesthetic perception.

Results

AesExpert (13B version) achieved the top position on the AesBench leaderboard, significantly outperforming commercial heavyweights. The model's overall score reached 89.50%, surpassing GPT-4V (80.80%) and Gemini-Pro-Vision (78.74%).

Its ability to resonate with human emotions (Aesthetic Empathy) showed a nearly 15% lead over baseline models, reaching 84.89%. Furthermore, its reasoning capabilities in "Aesthetic Interpretation" tasks improved precision by over 40% compared to its base model, LLaVA-1.5.

Conclusion and Future Work

AesExpert represents a significant step toward AI that perceives beauty with human-like nuance. While it excels with natural and AI-generated images, Artistic Images (AIs) remain a challenge due to their abstract nature and unconventional rules. Future research will focus on expanding the dataset to include broader artistic styles and exploring the temporal complexities of video aesthetics.

Summary updated

BibTeX
@inproceedings{huang2024aesexpert,
  author = {Yipo Huang and Xiangfei Sheng and Zhichao Yang and Quan Yuan and Zhichao Duan and Pengfei Chen and Leida Li and Weisi Lin and Guangming Shi},
  title = {{AesExpert: Towards Multi-modality Foundation Model for Image Aesthetics Perception}},
  booktitle = {ACM Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=fdAIgCCHXz},
}

Loading related content...

Figure