TL;DR
This paper introduces ArtFID, a novel quantitative metric for evaluating neural style transfer models. ArtFID addresses the critical lack of objective evaluation in this field, which currently relies heavily on subjective human judgment, by measuring both content preservation and style matching.
What Problem Does This Paper Address?
The field of neural style transfer (NST) has seen rapid advancements, leading to impressive visual synthesis results. However, a significant challenge remains: the absence of robust, quantitative evaluation metrics. Current methods primarily rely on qualitative assessments, which are subjective and difficult to reproduce. This makes it hard to objectively compare different NST models, understand their strengths and weaknesses, and track progress in the field. Unlike other areas of computer vision where standardized metrics are common, NST evaluation is still largely in the realm of visual inspection, hindering systematic research and development.
Methods
The core contribution of this paper is the ArtFID metric, designed to quantitatively assess the quality of neural style transfer. ArtFID is built upon two key components:
- Content Preservation: This measures how well the semantic content of the original image is retained in the stylized output. The authors leverage existing perceptual metrics like LPIPS (Learned Perceptual Image Patch Similarity) for this purpose, finding it to be slightly superior to alternatives like CLIP loss or VGG perceptual loss.
- Style Matching: This quantifies how well the style of the generated image matches the target style image. To achieve this, the authors first train a specialized Inception network on a large dataset of artworks. This network is designed with two classification heads to capture both artist and stylistic period information. The Inception features from the last layer of this network are then used to represent the style. The distance between the feature distributions of the style images and the stylized images is measured using the Fréchet distance (similar to how FID is computed for image generation).
These two measures are then combined multiplicatively to produce the final ArtFID score. A lower ArtFID score indicates better stylization performance. The paper also introduces an unbiased ArtFID computation method, addressing potential biases in FID calculations with finite sample sizes.
Datasets
- Training Dataset: A large-scale dataset of 250,000 labeled artworks was collected. This dataset includes unique labels for artists and stylistic periods, crucial for training the specialized Inception network for style representation. Images were sourced from public datasets like WikiArt, as well as museum databases and art collections.
- Evaluation Datasets:
- Content Images: Sampled from the Places365 dataset and the COCO dataset.
- Style Images: Sampled from the WikiArt dataset and the BAM dataset.
The use of multiple datasets for content and style images aims to increase the variability and robustness of the evaluation.
Key Insights and Ideas
- The "What is Style?" Problem: The paper acknowledges the philosophical difficulty in defining "style" and proposes a practical solution by learning image representations from a curated art dataset. This approach effectively captures stylistic similarities that are relevant for NST evaluation.
- Dual-Headed Inception Network: Training an Inception network with separate heads for artist and stylistic period allows for a richer representation of style, going beyond simple artist classification.
- ArtFID as a Complement, Not a Replacement: The authors emphasize that ArtFID is intended to complement existing qualitative evaluation methods, providing a much-needed objective benchmark rather than replacing human judgment entirely.
- Consistency with Human Judgment: A significant finding is the strong correlation between ArtFID scores and human preferences, validated through a large-scale user study. This suggests ArtFID effectively captures what humans perceive as good style transfer.
- Perturbation Robustness: ArtFID demonstrates consistency with increasing levels of image perturbations, similar to the FID metric, indicating its robustness to minor image variations.
Key Results
- Strong Correlation with Human Judgment: The ArtFID∞ metric achieved a Spearman's rank correlation coefficient of 0.939 with the human evaluation study, indicating a very strong agreement.
- Performance Comparison: Table 1 shows that ArtFID can effectively rank various style transfer models, often aligning with user study scores. For instance, PAMA [54] and IEContraAST [7] consistently rank higher across both metrics.
- Ablation Studies:
- Using ImageNet features for style matching resulted in a significantly lower correlation (0.329) compared to using the custom-trained art dataset features (0.939).
- Training the network with only artist labels or only style labels also reduced the correlation compared to using both.
- LPIPS was found to be slightly better than CLIP or VGG for measuring content preservation.
Limitations and Future Work
- Computational Expense: Obtaining accurate ArtFID measurements is computationally intensive, requiring significant processing for evaluating the Inception network and LPIPS metric on thousands of images.
- Future Work: The paper suggests that ArtFID can facilitate more systematic studies of style transfer methods, enabling researchers to analyze their strengths and weaknesses more effectively. The authors also hint at potential applications in other areas of visual synthesis.
Summary updated