Design-o-meter: Towards Evaluating and Refining Graphic Designs

Sahil Goyal, Abhinav Mahajan, Swasti Shreya Mishra, Prateksha Udhayanan, Tripti Shukla, Joseph K J, Balaji Vasan Srinivasan

TL;DR

This paper introduces Design-o-meter, a novel framework that automatically scores and refines graphic designs. It addresses the challenge of subjective design quality evaluation by using a data-driven approach, outperforming existing methods and even large multimodal models in design scoring.

What Problem Does This Paper Address?

Creating visually appealing and effective graphic designs is a complex, time-consuming process for designers, often involving iterative refinement based on feedback. While AI can now generate designs rapidly, evaluating their quality objectively and automatically remains a significant challenge. Existing metrics often fall short, failing to capture the nuances of aesthetic appeal and sometimes penalizing creative variations. This paper tackles the need for a robust, automated system that can not only score designs but also suggest improvements to enhance their aesthetic quality.

Methods

Design-o-meter consists of two main components: a Scorer and a Refiner.

  • Scorer: This module uses a data-driven, self-supervised approach based on a Siamese network. Instead of directly regressing a score, it learns to differentiate between "good" and "bad" designs through a contrastive setup. The scorer takes both the visual rendition of a design and a color-encoded layout map as input, allowing it to capture both visual aesthetics and structural layout principles. The training involves a combination of a ranking loss (hinge loss) and a similarity loss.
  • Refiner: This module employs a genetic algorithm to optimize designs. It uses the score from the Scorer as its objective function. The novelty here lies in a custom crossover operation called SWAN (Design Specific Crossover with Smart Snapping). SWAN intelligently combines elements from two parent designs by "snapping" elements into appropriate locations guided by grid lines and alignment principles, aiming to create a more aesthetically pleasing offspring.

Datasets

The primary dataset used for training and evaluation is the Crello dataset, which contains a large collection of design templates. To create training pairs for the scorer, the authors employ guided perturbations on existing designs to generate "bad" versions from "good" ones. These perturbations include adding noise to element positions and scales, and introducing clutter. For evaluation, they also use unbiased settings like "color" (where color schemes are altered) and "cross-match" (pairing good versions of one design with bad versions of another).

Key Insights and Ideas

  • Unified Scoring and Refining: The paper demonstrates the power of integrating design evaluation and refinement into a single, cohesive framework.
  • Beyond Pixel-Level Metrics: The success of Design-o-meter highlights the limitations of traditional metrics (like FID, mIOU) for capturing subjective design quality and the advantage of learning from comparative data.
  • Layout Encoding is Crucial: Incorporating a color-encoded layout map alongside the visual rendition significantly improves the scorer's ability to understand design principles like alignment and overlap.
  • SWAN's Smart Crossover: The novel SWAN crossover mechanism proves effective in navigating the complex design space, outperforming standard genetic algorithm crossover methods.
  • LLMs Aren't Always Best for Nuance: Despite their vast training data, large multimodal LLMs struggle to capture the subtle aesthetic differences that Design-o-meter can identify.

Key Results

  • Scorer Performance: Design-o-meter achieves the highest Rank Accuracy (RAcc) across multiple evaluation settings (biased, unbiased color, unbiased cross-match), significantly outperforming recent multimodal LLMs like GPT-4o and LLaVA-NeXT. For instance, in the biased setting, it achieved 94.97% RAcc compared to GPT-4o's 68.84%.
  • Refiner Performance: The refiner module, powered by SWAN, demonstrates state-of-the-art performance in both text box refinement (refine-text) and full design refinement (refine-all) tasks, achieving higher mIoU and lower mBDE than existing methods. For example, in the refine-all setting, it achieved a T-mIoU of 54.44%, significantly higher than the next best baseline at 50.08%.
  • Efficiency: The genetic algorithm-based refiner is computationally efficient, taking approximately 30 seconds to generate a refined design.

Limitations and Future Work

  • Dataset Bias: While the authors strive for unbiased evaluation, the training data is derived from existing designs, which might implicitly carry biases.
  • Perturbation Agnosticism: The current perturbations are largely agnostic to the dataset and focus on layout and scale. Future work could explore incorporating more sophisticated cognitive constraints into these perturbations.
  • Scorer as a Discriminator: The authors suggest that Design-o-meter's scorer could serve as an effective off-the-shelf discriminator for other layout and design generation frameworks.
  • Further Exploration of LLMs: While Design-o-meter outperforms LLMs in specific design evaluation tasks, exploring hybrid approaches that leverage LLMs' generative capabilities alongside Design-o-meter's scoring could be a promising direction.

Summary updated

Abstract

Graphic designs are an effective medium for visual communication. They range from greeting cards to corporate flyers and beyond. Off-late, machine learning techniques are able to generate such designs, which accelerates the rate of content production. An automated way of evaluating their quality becomes critical. Towards this end, we introduce Design-o-meter, a data-driven methodology to quantify the goodness of graphic designs. Further, our approach can suggest modifications to these designs to improve its visual appeal. To the best of our knowledge, Design-o-meter is the first approach that scores and refines designs in a unified framework despite the inherent subjectivity and ambiguity of the setting. Our exhaustive quantitative and qualitative analysis of our approach against baselines adapted for the task (including recent Multimodal LLM-based approaches) brings out the efficacy of our methodology. We hope our work will usher more interest in this important and pragmatic problem setting. Project Page: https://sahilg06.github.io/Design-o-meter/
BibTeX
@inproceedings{goyal2025design,
  author = {Sahil Goyal and Abhinav Mahajan and Swasti Shreya Mishra and Prateksha Udhayanan and Tripti Shukla and Joseph K J and Balaji Vasan Srinivasan},
  title = {{Design-o-meter: Towards Evaluating and Refining Graphic Designs}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00554},
}
Figure