ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services

Fengxian Ji, Jingpu Yang, Zirui Song, Lang Gao, Junhong Liang, Zhenhao Chen, Jinghui Zhang, Xiuying Chen

TL;DR

ServImage bridges the gap between academic image generation metrics and real-world commercial viability by grounding model evaluation in actual monetary outcomes. By introducing a dataset of over 1,000 paid design tasks worth more than $295k, the benchmark evaluates whether AI-generated images are actually "worth paying for." The results reveal a massive commercial viability gap between top proprietary models and open-source alternatives, showing that technical performance does not always translate to economic value.

What Problem Does This Paper Address?

Current image generation and editing benchmarks focus heavily on technical and perceptual metrics: instruction following, semantic controllability, realism, and physical commonsense. However, these academic metrics don't tell the whole story when it comes to commercial success.

In the real world, a design deliverable must satisfy strict business rules (resolution, format, policy compliance), align with specific brand styles, and fulfill a client's commercial intent. The fundamental question in a business context isn't just "Is the image good?" but "Is it worth paying for?" Prior to this paper, no benchmark evaluated models against market-validated value distributions and real human payment decisions.

Figure 1: Overview of the ServImage benchmark and evaluation framework. (a) We collect 1,070 paid design tasks from online crowdsourcing platforms and group them into Portrait, Product, and Digital categories. (b) Through the dataset pipeline with rule extraction/point mining, we obtain BRF, VEQ, and CNS scores to form ServImageScore. (c) Sixteen image generation and editing models produce about 33k candidate images. (d) Optionally, built on these scores, ServImageModel predicts human payment decisions. (e) We present representative tasks and model outputs as ServImage case studies. (f) Under the standard settlement scenario, we evaluate each model independently on the full task set and compare their commercial capability using the economic metric of total revenue.

Methods

The authors propose ServImageScore, an integrated evaluation framework that breaks down commercial acceptability into three core dimensions:

  1. Baseline Requirements Fulfillment (BRF): Measures explicit constraints from the client brief (e.g., size, format, specific elements) using a VLM-driven rule extraction pipeline.
  2. Visual Execution Quality (VEQ): Evaluates technical quality (clarity, artifacts), aesthetic quality (composition, color), and text quality.
  3. Commercial Necessity Satisfaction (CNS): Assesses consistency across multi-image tasks or image-editing workflows (e.g., preserving unedited regions).

To automate this, they build ServImageModel, a two-stage neural network utilizing a Concept Bottleneck Layer. Stage 1 predicts the three ServImageScore dimensions, and Stage 2 uses a task-aware payment head to output a calibrated acceptance probability, predicting whether a human client would approve and pay for the deliverable.

Datasets

  • ServImageBench: A newly introduced dataset consisting of 1,070 paid commercial design tasks collected from Chinese crowdsourcing platforms (Epwk.com and Zbj.com) between 2018 and 2025. It includes 2,046 human designer deliverables representing over $295k in contract value across Portrait, Product, and Digital categories.
  • Model Candidate Set: To evaluate AI models, the authors generated a candidate set of ~33k images using 16 mainstream text-to-image and editing models (12 proprietary, 4 open-source). Each image was annotated with ServImageScore dimensions and human payment decisions (accept/reject) using a double-annotation-with-adjudication protocol.
Figure 2: Task price distributions for Portrait, Product, and Digital categories. Dashed lines indicate median prices, showing a long-tailed pattern across categories.

Key Insights and Ideas

  • Technical Metrics Decouple from Economic Value: Models that score high on standard academic benchmarks often capture only a fraction of actual commercial value because they fail on strict business rules or specific brand alignment.
  • BRF is the Ultimate Gatekeeper: Failure analysis revealed that Baseline Requirements Fulfillment (BRF) is the most frequent cause of rejection (53.8%). If a model misses a specific text element or formatting constraint, it doesn't matter how beautiful the image is—the client won't pay.
  • Aesthetics Don't Drive Payment Boundaries: Surprisingly, the aesthetic quality dimension (VEQ-Aesthetic) barely separated accepted and rejected outcomes (Cohen's $d = -0.02$). Consistency and constraint satisfaction are far more critical for commercial viability than pure subjective beauty.
Composite score vs human acceptance rate.

Key Results

  • Proprietary vs. Open-Source Gap: Under a standard settlement scenario, proprietary models dominated. Gemini-Banana-Pro performed best, earning $243.98k (82.7% market share) of the total budget. The strongest open-source model, **Qwen-Image**, earned only $75.50k (25.6% share).
  • Crowdsourcing Competition: In a winner-takes-all setup simulating real crowdsourcing platforms, earnings were highly concentrated. Gemini-Banana-Pro took first place ($113.7k), followed by Gemini 2.5 Flash ($78.9k), while open-source models like flux-kontext-pro showed competitive promise ($37.9k).
  • Model Accuracy: ServImageModel (specifically the 4B-Think variant with LoRA tuning) achieved 82.00% accuracy in predicting human payment decisions, significantly outperforming general-purpose VLM baselines.
  • Metric Alignment: ServImageScore demonstrated substantially better alignment with human payment decisions (higher AUC, Spearman rank correlation, and PR-AUC) than seven existing generation and editing metrics.
Figure 5: Metric comparison on the test set. Bars show AUC, Spearman, and PR-AUC for seven metrics and ServImageScore, higher is better.

Limitations and Future Work

  • Geographic and Platform Bias: The dataset is sourced entirely from two Chinese crowdsourcing platforms, meaning pricing norms, task distributions, and aesthetic preferences might reflect regional biases.
  • Binary Simplification: Commercial acceptability is treated as a binary accept/reject decision. Real-world design workflows are highly iterative, involving partial acceptance, revisions, and multi-turn negotiations.
  • Future Directions: Future work aims to expand the benchmark to include multi-turn evaluation, revision-based workflows, and multi-judge ensembles to improve robustness and generalizability.

Summary updated

Abstract

Recent image generation and editing models demonstrate robust adherence to instructions and high visual quality on academic benchmarks. However, their performance on paid, real-world design projects remains uncertain. We introduce \textbf{ServImage}, a benchmark that explicitly correlates model outputs with economic value in commercial design projects. ServImage consists of (i) \textbf{\textit{ServImageBench}}: a dataset of 1.07k paid commercial design tasks and 2.05k designer deliverables totaling over \$295k, covering portrait, product, and digital content, along with 33k candidate images and 33k human annotations. (ii) \textbf{\textit{ServImageScore}}: an integrated scoring system that combines three quality dimensions: baseline requirements fulfilment, visual execution quality, and commercial necessity satisfaction. These three dimensions are designed to characterize the factors that drive human payment decisions and indicate whether an image is commercially acceptable. (iii) \textbf{\textit{ServImageModel}}: under this scoring system, we propose a payment prediction model trained on the human-annotated candidate images, achieving 82.00\% accuracy in predicting human payment decisions and producing calibrated payment probabilities. ServImage provides a comprehensive foundation for assessing the commercial viability of image generation models and offers a scalable resource for future research on economically grounded vision systems \href{https://github.com/FengxianJi/ServImage}{Github.}
BibTeX
@inproceedings{ji2026servimage,
  author = {Fengxian Ji and Jingpu Yang and Zirui Song and Lang Gao and Junhong Liang and Zhenhao Chen and Jinghui Zhang and Xiuying Chen},
  title = {{ServImage: An Image Generation and Editing Benchmark from Real-world Commercial Imaging Services}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2014},
}
Figure