TL;DR
ArtScore is a novel, reference-free metric designed to quantify the "artness" of AI-generated images at the instance level. By training a neural network on a synthetic dataset of interpolated transitions from photos to paintings, the authors created a tool that aligns significantly better with human aesthetic judgment than traditional metrics like Gram loss or FID.
What Problem Does This Paper Address?
As AI image generation explodes, we lack a reliable way to answer a simple question: How "artistic" is this specific image?
Existing evaluation tools have major gaps:
- Distribution-based metrics (like FID or Inception Score) require large batches of images and can't evaluate a single result in isolation.
- Reference-based metrics (like SSIM or LPIPS) require a "ground truth" image to compare against, which doesn't exist for unconditional generation.
- Stylistic metrics (like Gram loss) often focus so much on texture that they ignore structural artifacts that make an image look "fake" or unappealing to humans.
The authors propose ArtScore to fill this gap, providing a reference-free, instance-level score that captures the nuanced qualities of fine-art paintings—such as composition, brushwork, and harmony—rather than just matching textures.
Methods
The researchers developed ArtScore through a clever three-step pipeline that bypasses the need for expensive human labeling:
- StyleGAN Adaptation: They took pre-trained StyleGAN2 models (optimized for photorealism in domains like faces, horses, and landscapes) and adapted them to artistic styles using transfer learning. They employed two protocols: FreezeG (tuning only later layers) and few-shot adaptation (using a small cluster of representative paintings).
- Interpolated Image Generation: By linearly interpolating the weights between the photorealistic "base" model and the artistic "adapted" model, they generated sequences of images. Each sequence features the same content but transitions smoothly from a photo to a painting. The interpolation weight ($\alpha$) serves as a "pseudo-annotation" for the artness level.
- ArtScore Training: They trained a ResNet50-based model to predict these artness levels. Crucially, they used a learn-to-rank objective (ListMLE loss) rather than simple regression. This allows the model to learn the relative artness within a sequence, which is more robust than trying to assign an arbitrary absolute value to "art."
Datasets
The study utilized several foundational datasets to build its training and evaluation pipelines:
- WikiART: A massive collection of over 80,000 paintings used to adapt the StyleGAN models to artistic styles.
- FFHQ, LSUN Horse, and LHQ: Used as the source for photorealistic domains (faces, horses, and landscapes, respectively).
- The Interpolated Artness Dataset: A new dataset created by the authors containing 143,742 sequences (over 1.7 million images), each spanning 12 levels of artness from photo to art.
Key Insights and Ideas
- Interpolation as Supervision: The most elegant idea is using model weight interpolation to generate "ground truth" labels. It provides a continuous, controllable spectrum of artness that human annotators would find impossible to label consistently.
- Ranking > Regression: The authors found that training the model to rank images by artness produced better results than asking it to predict a fixed score. This reflects the subjective nature of art; while we might not agree on a "10/10," we can usually agree that Image A looks more like a painting than Image B.
- The Gram Loss Paradox: The paper confirms a frustrating reality in Neural Style Transfer (NST): lower Gram loss (which mathematically means better style matching) often correlates negatively with human preference. Gram loss can be "tricked" by textures that humans find messy or full of artifacts.
- Cross-Sequence Swapping: To make the model robust, the authors randomly swapped images between different sequences during training. This forced the model to compare the artness of a horse painting to a landscape painting, preventing it from just memorizing content-specific cues.
Key Results
- Human Alignment: ArtScore achieved a Spearman’s rank correlation of 0.6364 with human artistic evaluation, significantly outperforming Gram loss (-0.3286) and standard content metrics.
- Synergy with Existing Metrics: When ArtScore was combined with ArtFID (a distribution-based metric), the correlation with human judgment jumped to 0.8196, suggesting that ArtScore captures unique aesthetic information that other metrics miss.
- Instance-Level Success: In user studies, ArtScore correctly predicted which image a human would prefer as "more artistic" with 64.33% accuracy for images with the same content, and remained consistent even when comparing images with completely different subjects.
Limitations and Future Work
While ArtScore is a significant step forward, the authors acknowledge some hurdles:
- Subjectivity: "Artness" is inherently subjective. The model reflects the general aesthetic trends found in the WikiART dataset and the preferences of the user study participants, which may not align with every artistic subculture.
- Domain Specificity: The current model is trained on faces, horses, and landscapes. Its performance on abstract art or highly complex scenes not covered in the training domains remains an open question.
- Future Integration: The authors suggest that ArtScore could eventually be used as a loss function or a guidance function for diffusion models, essentially acting as an "automated art critic" that nudges the AI toward more aesthetically pleasing results during the generation process.
Summary updated