Visual Interestingness Decoded: How GPT-4o Mirrors Human Interests

Fitim Abdullahu, Helmut GrabnerICCV 2025

TL;DR

This paper investigates how well Large Multimodal Models (LMMs), specifically GPT-4o, understand and predict "visual interestingness" – a subjective concept crucial for engaging content. They found that GPT-4o shows significant alignment with human judgments, outperforming existing methods and offering a promising avenue for automated content analysis and dataset creation.

What Problem Does This Paper Address?

In today's data-saturated world, capturing and holding user attention is paramount. "Visual interestingness" is the key to this, but it's a notoriously subjective and fuzzy concept. While LMMs have shown impressive capabilities in understanding and generating content, their ability to grasp subjective qualities like interestingness remains an open question. Manually annotating vast datasets for interestingness is costly and time-consuming. This paper aims to bridge this gap by exploring whether LMMs can effectively capture visual interestingness, thereby reducing the need for human annotation and providing deeper insights into what makes an image compelling.

Methods

The researchers employed a multi-pronged approach:

  • Single Image Assessment: They collected human judgments on whether individual images were interesting, along with brief explanations. This was then compared to GPT-4o's responses to the same prompt.
  • Pairwise Comparison: To overcome the "always interesting" bias observed in single-image assessments, they presented GPT-4o with pairs of images and asked it to identify the more interesting one. This approach is more robust for subjective comparisons.
  • Learning-to-Rank Model: They trained a computational model using a Siamese network architecture. This model learned to rank images based on the annotations (both human and GPT-4o) to predict interestingness.
  • Analysis of Explanations: The study delved into the textual explanations provided by both humans and GPT-4o to understand the reasoning behind their judgments and identify areas of agreement and divergence.
  • Hierarchical Clustering: Embeddings of image descriptions were clustered to reveal semantic groups and analyze agreement/disagreement patterns.

Datasets

  • Flickr-User Dataset: The study utilized 1,000 images sampled from the Flickr-User dataset, chosen for their representative and diverse content.
  • Newly Introduced Dataset: A dataset of 1,000 images was curated, resulting in 2,500 image pairs for the relative interestingness assessment. This dataset includes metadata and multiple verified labels from human annotators and LMMs.

Key Insights and Ideas

  • GPT-4o as a Proxy for Human Interest: GPT-4o demonstrates a remarkable ability to mirror human judgments on visual interestingness, especially when humans themselves agree. This suggests LMMs can serve as valuable, albeit imperfect, proxies for human perception.
  • The "Always Interesting" Bias: In single-image assessments, both humans and LMMs tend to label most images as interesting, likely due to the prompt's framing. This highlights the importance of comparative tasks for more nuanced analysis.
  • Systematic Bias in GPT-4o: The study identified a systematic bias in GPT-4o, where it consistently favored the second image in a pair, regardless of content. This is a crucial limitation to consider when using its outputs.
  • Demographics Don't Matter (for Everyday Images): Interestingly, when tested with everyday images, GPT-4o's judgments were not significantly influenced by simulated demographic profiles (gender, age, continent). However, this might change with more specialized image sets.
  • Explanations Reveal Nuances: Analyzing the "why" behind judgments revealed that while humans often use simple, repetitive terms, GPT-4o provides more elaborate descriptions, sometimes highlighting aspects humans might overlook (e.g., the story in a cluttered scene).

Key Results

  • High Agreement with Humans: GPT-4o achieved a 66.2% agreement with human judgments on pairwise image interestingness, rising to 73.8% when human consensus was high.
  • Outperforming Existing Models: A learning-to-rank model trained on GPT-4o annotations achieved higher accuracy and Spearman rank correlation than models trained on human annotations or other established methods for visual interestingness and related concepts (aesthetics, memorability, social metrics).
  • Aesthetic and Social Metrics Align: Models focused on aesthetics and social metrics (views, favorites, comments) showed better alignment with human and GPT-4o judgments compared to memorability.
  • GPT-4o Superiority: GPT-4o consistently outperformed other LMMs (Llama 3.2, DeepSeek-VL2) in terms of annotation consistency and agreement with human judgments, although these models were tested with workarounds due to input limitations.

Limitations and Future Work

  • Systematic Bias: The identified systematic bias in GPT-4o favoring the second image in a pair is a significant limitation that needs to be addressed for reliable large-scale annotation.
  • Limited Demographic Testing: While initial demographic tests showed no impact on everyday images, a more extensive study with diverse image types and demographic profiles is warranted.
  • Domain Gap: The study's reliance on Flickr images might limit generalizability to other domains. Future work should explore datasets beyond this platform.
  • Further Investigation of Gaps: Understanding the subtle differences and divergences between human and LMM judgments, especially concerning demographic factors, remains an important area for future research.
  • Knowledge Distillation: The paper suggests that LMMs, particularly GPT-4o, hold significant potential for knowledge distillation to create more efficient models for visual interestingness prediction.

Summary updated

BibTeX
@inproceedings{abdullahu2025visual,
  author = {Fitim Abdullahu and Helmut Grabner},
  title = {{Visual Interestingness Decoded: How GPT-4o Mirrors Human Interests}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure