TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization

Mingyue ChengarXiv 2024 · DOI · arXiv

TL;DR

This paper introduces TextMatch, a novel framework that significantly improves the alignment between text prompts and generated images in text-to-image (T2I) generation and editing tasks. By leveraging large language models (LLMs) for multimodal optimization, TextMatch dynamically refines prompts through iterative visual scoring and chain-of-thought reasoning, leading to more accurate and high-fidelity image outputs.

What Problem Does This Paper Address?

Current text-to-image (T2I) generative models, despite their impressive capabilities, often struggle with accurately capturing complex prompts that involve intricate details like attribute binding, spatial relationships, and numerical accuracy. This leads to generated images that may not align with user intent, contradict prompts, or fail to compose multiple elements cohesively. Existing methods often lack a robust mechanism to ensure semantic consistency between the prompt and the generated output, especially when dealing with nuanced or complex instructions.

Methods

TextMatch tackles this challenge through a multimodal optimization framework with three core components:

  1. T2I or Image Editing Model: The underlying generative model (e.g., SDXL, DALL-E 3) that produces the image.
  2. Visual Scoring Mechanism: This component evaluates the consistency between the generated image and the user prompt. It comprises:
  3. Prompt-Inspired Question Synthesizer (PIQS): An LLM that generates targeted questions and answers based on the user prompt and extracted elements (objects, attributes, spatial relations, etc.). For image editing, it also considers the base image.
  4. Visual Cognition Assessor (VCA): A Visual Question-Answering (VQA) model that answers the PIQS-generated questions based on the image, providing a quantitative score of fidelity.
  5. MLLM-Driven Dynamic Prompt Optimization: A large multimodal language model (MLLM) acts as a prompt optimizer. It uses:
  6. Multimodal In-Context Learning: The MLLM receives context including the original prompt, generated images, previous prompts, and visual scoring results.
  7. Chain of Thought (CoT) Planning: The MLLM employs CoT reasoning to identify weaknesses in prompt-image alignment and applies operations like "Add," "Enhance," or "Modify" to refine the prompt iteratively. This process aims to improve accuracy and precision.

The framework operates in iterative rounds, refining prompts until a satisfactory score is achieved or a maximum number of iterations is reached.

Datasets

The paper evaluates TextMatch on two benchmarks:

  • T2I-CompBench: Used for text-to-image generation tasks, focusing on complex prompts involving multiple objects, attributes, and relationships.
  • MagicBrush: Used for image editing tasks, featuring various types of text instructions for editing.

The authors do not introduce new datasets.

Key Insights and Ideas

  • LLMs as Dynamic Prompt Refiners: The core innovation is using LLMs not just to generate text but to actively refine prompts based on visual feedback, bridging the gap between textual intent and visual output.
  • Iterative Refinement is Crucial: TextMatch's strength lies in its iterative process. By repeatedly scoring and optimizing, it can correct subtle errors and improve alignment over multiple steps, which is particularly effective for complex prompts.
  • Visual Scoring as a Bridge: The PIQS and VCA combination provides a structured, quantitative way to assess image-text consistency, offering actionable feedback for the LLM optimizer.
  • CoT Enhances Optimization: Chain-of-Thought reasoning allows the LLM to break down the problem, identify specific areas of misalignment, and make more targeted prompt adjustments.
  • Multimodal Context is Key: Incorporating image context alongside textual information significantly improves the LLM's ability to understand and correct errors in generated images.

Key Results

  • T2I Generation: TextMatch significantly outperforms state-of-the-art T2I models like SDXL and DALL-E 3 on the T2I-CompBench benchmark across various metrics, including attribute binding, numeric accuracy, and complex relationships. For example, TextMatch with SDXL shows improvements of +0.0474 in Color, +0.0216 in Shape, and +0.0074 in Non-Spatial attributes compared to SDXL alone. Similarly, TextMatch with DALL-E 3 shows notable gains, especially in Object Relationship (+0.0318) and Complex (+0.0680) metrics.
  • Image Editing: TextMatch demonstrates superior performance in text-guided image editing tasks on the MagicBrush benchmark, outperforming models like InstructPix2Pix and HIVE across multiple metrics (L1, L2, CLIP-I, DINO, CLIP-T). It shows particular strength in handling color and attribute binding with high precision while maintaining image integrity.
  • Iterative Improvement: Experiments show that prompt-image consistency improves with iterative rounds, peaking around three iterations before potentially declining due to MLLM limitations.

Limitations and Future Work

  • Time Complexity: The iterative nature of TextMatch can be time-consuming. The authors aim to optimize the prompt refinement process to reduce the number of iterations required.
  • T2I Model Dependency: The quality of generated images is still dependent on the capabilities of the underlying T2I model.
  • Future Work: The authors plan to explore integrating agents for automated T2I model selection and further enhance the quality of generated images. They also aim to leverage advanced T2I models within their framework.

Summary updated

Abstract

Text-to-image generative models excel in creating images from text but struggle with ensuring alignment and consistency between outputs and prompts. This paper introduces TextMatch, a novel framework that leverages multimodal optimization to address image-text discrepancies in text-to-image (T2I) generation and editing. TextMatch employs a scoring strategy powered by large language models (LLMs) and visual question-answering (VQA) models to evaluate semantic consistency between prompts and generated images. By integrating multimodal in-context learning and chain of thought reasoning, our method dynamically refines prompts through iterative optimization. This process ensures that the generated images better capture user intent of, resulting in higher fidelity and relevance. Extensive experiments demonstrate that TextMatch significantly improves text-image consistency across multiple benchmarks, establishing a reliable framework for advancing the capabilities of text-to-image generative models. Our code is available at https://anonymous.4open.science/r/TextMatch-F55C/.
BibTeX
@misc{cheng2024textmatch,
  author = {Mingyue Cheng},
  title = {{TextMatch: Enhancing Image-Text Consistency Through Multimodal Optimization}},
  year = {2024},
  eprint = {2412.18185},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.18185},
}
Figure