StyleJudge: A Human-Validated, In-Context-Learning Vision-Language Evaluator for Artistic Style Transfer

Lokesh, Pramukto Mandal, Somanshu Juneja, Jashnoor Singh, Amit Kumar, Pranav Adlinge, Richa Thakur, Kunal Swami, Aditya NigamarXiv 2026 · arXiv

TL;DR

StyleJudge is a compact, 0.8-billion-parameter vision-language model designed to evaluate artistic style transfer by "looking" at reference examples before judging a new image. By using in-context learning and knowledge distillation from a massive 72B teacher model, it achieves significantly higher correlation with human judgment than traditional metrics like CLIP or Gram-matrix loss.

What Problem Does This Paper Address?

Artistic Style Transfer (AST)—the process of redrawing a "content" image in the style of a "reference" artwork—has advanced rapidly thanks to diffusion models. However, evaluating these models is still a mess.

Traditional metrics like Gram-matrix loss (which looks at texture statistics) or CLIP similarity (which looks at semantic alignment) operate in a "zero-shot" vacuum. They score an image pair without ever seeing what the intended style transformation is actually supposed to look like. Consequently, these metrics often reward images that look nothing like the target style or penalize successful abstractions because they "drift" too far from the original content. Humans, by contrast, calibrate their expectations by looking at examples of a style before judging a candidate. StyleJudge aims to bridge this gap by bringing that same "calibration" step to automated evaluation.

Methods

The researchers introduced several novel components to build StyleJudge:

  1. In-Context Learning (ICL): Instead of judging a query pair in isolation, the model is prompted with two "perfect" exemplar pairs (source $\rightarrow$ stylized) of the target style. This turns evaluation into a visual analogy task.
  2. Knowledge Distillation: To make the system practical, the authors distilled the reasoning capabilities of a massive Qwen2.5-VL 72B teacher model into a tiny Qwen3.5-0.8B student. This allows the evaluator to run on consumer-grade hardware (like an NVIDIA T4) with minimal memory.
  3. Graded Supervision via Morphing: To teach the model to recognize varying intensities of stylization, they didn't just use "styled" vs. "unstyled" images. They adapted a tool called FreeMorph to create 10-frame "graded strips" showing a smooth transition from the source to the final stylized result.
  4. Composite Scoring Logic: The model outputs three scores (1–10): Style Fidelity (SF), Content Preservation (CP), and Rendering Quality (RQ). Crucially, they use a "clamping" logic: if the content is destroyed or the rendering is full of artifacts (score $\leq 3$), the final score is capped, reflecting how human raters treat these as "blocking conditions."

Datasets

  • Source Pool: ~10,000 images curated from seven public collections (including LAION, HQ-50K, and iNaturalist) covering landscapes, portraits, and macro photography.
  • Artistic Styles: 12 distinct styles (e.g., Cubism, Mosaic, Charcoal, Watercolor) generated using FLUX.1-dev and style-specific LoRAs.
  • Human-Correlation Benchmark: A newly released dataset of 878 human-rated pairs. This serves as the "gold standard" for testing how well automatic metrics actually align with real human preferences.

Key Insights and Ideas

  • Context is the Missing Ingredient: The paper demonstrates that zero-shot models often collapse into generic "image quality" assessors. By providing exemplars, the model learns to ignore irrelevant features and focus on specific "style markers" (like the grout lines in a mosaic or the impasto texture in an acrylic painting).
  • The "Clamping" Heuristic: One of the most intuitive findings was that humans don't "trade off" content for style. If an image is a garbled mess, it doesn't matter how "stylized" it is—humans rate it poorly. Incorporating this threshold-based logic into the final score calculation was vital for human alignment.
  • Distillation Efficiency: Surprisingly, the 0.8B student retained most of the 72B teacher's alignment. The student actually produced 67.8% shorter reasoning traces, making it faster and more focused while maintaining a similar correlation to human ratings.

Key Results

  • Superior Correlation: StyleJudge reached a Pearson correlation of 0.291 on seen styles and 0.381 on unseen styles. For comparison, Gram-matrix loss ($r=0.041$) and CLIP ($r=0.168$) were nearly uncorrelated, while LPIPS actually showed a negative correlation ($r=-0.404$) because it penalizes the very changes that define style transfer.
  • Generalization: The model performed better on "unseen" styles (styles it wasn't trained on) than on seen ones, suggesting it learned a general procedure for evaluation rather than just memorizing specific textures.
  • Efficiency: It can score a pair in roughly 39 seconds using only 2.5 GB of VRAM, making it a viable tool for researchers to use during model development.

Limitations and Future Work

  • Style-Level Calibration: While the model ranks images well within a style, its "mean" score varies between styles (e.g., it might be naturally "stingier" with Pop Art than with Watercolor). Future work could involve per-style calibration.
  • Rater Diversity: The human benchmark was created by three project-team members. A broader, more diverse panel of art experts would provide a more robust "human ceiling."
  • Generator Bias: All images were generated using the FLUX family. Testing how well StyleJudge evaluates outputs from other architectures (like GANs or different diffusion backbones) is a logical next step.

For those interested in trying it out, the authors have released the model weights, the human-rating benchmark, and their SDXL-adapted FreeMorph code on GitHub.

Summary updated

Abstract

Automatic evaluation of artistic style transfer remains an open problem: existing metrics score an image pair in isolation, without seeing an example of the intended style transformation, and correlate poorly with human judgment. We introduce StyleJudge, a compact vision-language evaluator conditioned on a small set of reference source-to-stylized pairs before scoring a query pair, mirroring how a human rater calibrates to a style before judging a candidate. On a human-annotated benchmark of 878 rated pairs across 12 artistic styles (728 from 10 training/validation styles, 150 from 2 unseen styles), our efficient 0.8B-parameter StyleJudge reaches a Pearson correlation of 0.291 (Spearman 0.325) with human ratings on seen styles and 0.381 on unseen ones. On seen styles, it achieves the highest positive correlation among tested metrics: Gram-matrix style loss and CLIP similarity correlate weakly ($r=0.041$, $r=0.168$), while LPIPS (scored higher when closer to the source) correlates negatively ($r=-0.404$). We achieve this by distilling evaluation knowledge from a 72B-parameter vision-language teacher used only during training, which reaches $r=0.327$, confirming that most of its human alignment survives compact distillation. For training, we build a dataset of source-to-stylized pairs across the 12 styles via LoRA-adapted diffusion, densified with graded intermediate examples from a high-resolution image-morphing pipeline so the evaluator recognizes partial and complete stylization. On two held-out validation styles, StyleJudge reaches 28.2% exact-match agreement with its teacher's structured scores versus 1.1% for the untuned base model, with 67.8% shorter reasoning traces, and scores a pair in 39 s (median) using 2.5 GB of GPU memory on one NVIDIA T4. We release the human-rating benchmark, annotation guideline, SDXL-adapted FreeMorph code, and model weights.
BibTeX
@misc{lokesh2026stylejudge,
  author = {Lokesh and Pramukto Mandal and Somanshu Juneja and Jashnoor Singh and Amit Kumar and Pranav Adlinge and Richa Thakur and Kunal Swami and Aditya Nigam},
  title = {{StyleJudge: A Human-Validated, In-Context-Learning Vision-Language Evaluator for Artistic Style Transfer}},
  year = {2026},
  eprint = {2611.19475},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19475},
}
Figure