Aesthetically Relevant Image Captioning

Zhipeng Zhong, Fei Zhou, Guoping QiuAAAI 2023 · DOI

Introduction

In computer vision, evaluating the beauty of an image has traditionally been split into two distinct tasks: Aesthetic Quality Assessment (AQA), which assigns a numerical score to an image, and Image Aesthetic Captioning (IAC), which generates descriptive text. However, these fields have historically remained siloed, leading to a gap between knowing that an image is beautiful and understanding why.

Existing IAC models often struggle with "noisy" datasets. Most user-generated comments on photography platforms are social or generic (e.g., "Great shot!") rather than aesthetically descriptive. Consequently, models trained on this data learn to produce irrelevant output. To address this, researchers have introduced Aesthetically Relevant Image Captioning (ARIC), a framework supported by a massive new dataset and a novel relevance metric.

The DPC2022 Dataset

To facilitate research in this area, the authors present DPC2022, the largest dataset of its kind. It comprises 510,000 images sourced from photography competition websites, paired with over 5 million comments and 350,000 aesthetic ratings on a 1–10 scale. This dataset provides the necessary scale and quality to move beyond the limitations of smaller predecessors like AVA or PCCD.

Distribution of aesthetic scores, comments, sentence lengths, and ARS in the DPC2022 dataset.

As shown in the data distributions, the dataset captures a wide range of aesthetic scores and comment lengths, providing a rich foundation for training models that understand both the visual and textual aspects of photography.

Methodology: Quantifying Aesthetic Relevance

The core of the approach is the Aesthetic Relevance Score (ARS). This metric quantifies how descriptive a sentence is regarding an image's artistic merits. It integrates five key factors: presence of aesthetic keywords (e.g., "composition"), sentence length, absence of generic object labels, sentiment analysis, and term frequency (tf-idf). The researchers developed a BERT-based ARS Predictor that achieves a Spearman’s rank correlation (SRCC) of over 0.95, demonstrating that aesthetic relevance can be modeled mathematically with high reliability.

The ARIC Model Architecture

The ARIC model leverages the ARS to improve captioning quality through two primary mechanisms: weighted loss and diverse selection.

The ARIC model architecture featuring a bottom-up-attention image encoder and a multi-layered decoder with meshed connections.

Weighted Loss

During training, the model employs an ARS-weighted cross-entropy loss. This ensures that the system prioritizes learning from highly relevant aesthetic critiques while down-weighting generic social comments.

Diverse Aesthetic Caption Selector (DACS)

At the inference stage, the model generates multiple candidate captions. Instead of simply picking the most probable one—which often leads to repetitive results—the DACS clusters candidates by similarity and uses the ARS Predictor to select the most relevant and diverse set.

Key Results and Insights

The experiments yielded several significant findings:

  • AQA Performance: User reviews were found to be highly informative. Predicting an image's aesthetic score using only the text of human comments (SRCC 0.833) was more accurate than using the image itself (SRCC 0.675). This validates that high-ARS text captures the "essence" of visual quality.
  • Captioning Quality: The ARIC model outperformed baselines across standard metrics like METEOR, CIDEr, and SPICE. Qualitatively, the model successfully generates specific critiques, such as discussing color contrast rather than just offering praise.
  • Diversity: The DACS mechanism effectively mitigated "mode collapse," ensuring the model produces varied descriptions for different images rather than repeating a few safe phrases.

Conclusion and Future Work

The ARIC framework demonstrates that bridging the gap between scoring and describing beauty is possible through rigorous relevance modeling. While a gap remains between human-level critiques and AI-generated text, particularly in perceiving fine-grained artistic details, this work provides a robust foundation for future vision-language integration in the aesthetic domain.

Summary updated

BibTeX
@inproceedings{zhong2023aesthetically,
  author = {Zhipeng Zhong and Fei Zhou and Guoping Qiu},
  title = {{Aesthetically Relevant Image Captioning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i3.25485},
}
Figure