VISUAL PROMPTING METHODS FOR GPT-4V BASED ZERO-SHOT GRAPHIC LAYOUT DESIGN GENERATION

Kunal Singh, Mukund Khanna, Ankan Biswas, Pradeep Moturi, ShivamICLR 2024 · OpenReview

Graphic design is a delicate dance between aesthetics and information. For an AI, placing a "Happy Birthday" string on a busy background isn't just about finding empty space; it’s about understanding visual hierarchy, avoiding the occlusion of key subjects (like a face or a logo), and maintaining readability. While specialist models have tried to automate this, a new paper from researchers at Fractal Analytics suggests that a generalist model like GPT-4V can do the job—if you give it the right visual cues.

The Spatial Reasoning Gap

Automated graphic layout generation is a complex multi-modal challenge. A model must simultaneously understand the semantic content of an image (e.g., "this is a person's face") and the geometric constraints of the canvas.

Existing approaches usually fall into two camps: generative models that often ignore the actual text content, or object-detection-style models that struggle with aesthetic placement. The authors identify a specific gap: Large Multimodal Models (LMMs) like GPT-4V are brilliant at understanding what is in an image, but they are notoriously bad at pinpointing where things are in terms of exact pixel coordinates. This "spatial reasoning gap" makes them unreliable for professional-grade design tasks out of the box.

Methods: Augmenting AI with Visual Prompts

The core innovation of this research is Visual Prompting. Instead of just sending a raw image to GPT-4V, the researchers augmented the images with visual guides to help the model anchor its spatial understanding.

The visual prompting pipeline showing an input image augmented with numbered regions before being processed by GPT-4V.

They tested several flavors of these prompts to see which best improved the model's spatial reasoning:

  • Set-of-Marks (SoM): Using external segmentation models to partition the image into meaningful regions and labeling them.
  • Grid Segmentation: Dividing the image into a simple 3x3 grid.
  • Superpixel Segmentation (SLIC & Watershed): Using traditional computer vision algorithms to group pixels into 9 semantically similar clusters and marking their centers numerically.
  • Reference Image: A one-shot approach where the model is shown a "perfect" example of a similar layout before being asked to design a new one.

The model was then prompted to output a specific JSON-like format containing font size, color (black or white for contrast), and bounding box coordinates.

The Problem with "Dirty" Datasets

During their evaluation, the researchers took a stand against flawed benchmark data. They found that the standard dataset used in prior layout generation research (from the LayoutDetr paper) relied on diffusion-based inpainting to remove text from original designs.

Comparison of an original design and its inpainted version, highlighting the blurry artifacts that confuse AI models.

This inpainting process left behind blurry artifacts. GPT-4V, acting intelligently, "rightly avoided" placing text over those messy regions, which led to artificially poor evaluation scores. To fix this, the team curated a new dataset of 50 high-quality images from Canva across three categories (greeting cards, invitations, and advertisements). These images were manually edited to ensure no artifacts remained, providing a clean ground truth for evaluation.

Key Insights and Results

The researchers used Intersection over Union (IoU) to measure how well the AI's predicted text box matched the human-designed one. The results revealed several fascinating insights:

  • The Semantic-Spatial Paradox: GPT-4V often knows exactly where the text should go semantically (e.g., "in the empty space to the left of the coffee cup"), but fails to translate that into correct [x, y] coordinates without help.
  • Simplicity Wins: Interestingly, the simpler Grid and SLIC methods performed the best. A simple, consistent spatial reference is often more effective for the model's internal reasoning than complex, high-resolution segmentation.
  • Beating the Specialists: The baseline GPT-4V achieved a modest IoU of 0.068. However, with the Grid prompt, it jumped to 0.125 (and a "Best of 3" score of 0.192). The SLIC method reached 0.107 (Best of 3: 0.186). In comparison, the specialized, supervised model LayoutDetr scored 0.165.

Ultimately, with Grid or SLIC visual prompts, a zero-shot generalist model (GPT-4V) successfully matched or surpassed the performance of a specialist model trained specifically for this task.

Limitations and Future Work

Despite the impressive improvements, the authors note that GPT-4V remains non-deterministic, meaning it can yield different answers for the exact same prompt, leading to variance in design quality. Furthermore, while visual prompting helps significantly, the model still occasionally struggles with the exact "tightness" of bounding boxes.

Future work could involve more granular visual prompts or "chain-of-thought" prompting, where the model first identifies the region by label and then calculates the coordinates in a secondary step. For now, this paper proves that if you want an AI to be a better designer, you just need to give it a better map.

Summary updated

Abstract

Graphic layout design generation is a challenging problem in computer vision. The key aspect of the challenge is ensuring coherent placement of textual elements on the background image to ensure aesthetic appeal and avoiding occlusion of key visual elements. Although prior methods have made attempts to solve this multi-modal problem, they couldn't perfect it. Owing to the complexity required in understanding the relationship between visual and text elements in the aforementioned task, we investigate GPT-4-Vision(GPT-4V), a large multimodal models(LMMs), to do zero-shot graphic layout design generation in a versatile manner. Our approach explores various off-the-shelf segmentation/superpixel methods to identify and mark the key regions to visually augment the image to enhance GPT-4V's spatial reasoning capability . The results of our comprehensive experiments on a self-curated dataset demonstrates the efficacy of our proposed visual prompting methods, showing improvement over standard GPT-4V prompting method and also performing at par and even better, for some techniques, than state-of-the-art specialist model.The code and data is available at https://anonymous.4open.science/r/VISUAL-PROMPTING-TECHNIQUES-FOR-GPT-4V-BASED-ZERO-SHOT-GRAPHIC-LAYOUT-DESIGN-GENERATION-5A6E
BibTeX
@inproceedings{singh2024visual,
  author = {Kunal Singh and Mukund Khanna and Ankan Biswas and Pradeep Moturi and Shivam},
  title = {{VISUAL PROMPTING METHODS FOR GPT-4V BASED ZERO-SHOT GRAPHIC LAYOUT DESIGN GENERATION}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=DI4gETm7aa},
}

Loading related content...

Figure