Generating Fine Details of Entity Interactions

Xinyi Gu, Jiayuan Mao

TL;DR

This paper introduces "DetailScribe," a novel framework that significantly improves text-to-image generation for complex scenes with intricate entity interactions. By leveraging multimodal LLMs for concept decomposition and iterative refinement, DetailScribe generates more accurate and detailed images than existing state-of-the-art models, addressing a key limitation in current T2I capabilities.

What Problem Does This Paper Address?

Modern text-to-image (T2I) models excel at generating realistic images of individual objects or simple scenes. However, they often struggle with prompts that describe complex interactions between multiple entities or intricate spatial arrangements. This limitation stems from a lack of comprehensive datasets and benchmarks specifically designed to evaluate and train for these fine-grained interactions. Existing benchmarks tend to focus on single objects or basic spatial relationships, leaving a gap in assessing and improving a model's ability to capture nuanced, multi-entity dynamics.

Methods

DetailScribe employs a "generate-then-refine" pipeline augmented by multimodal large language models (MLLMs). The process involves three main stages:

  1. Concept Decomposition: An LLM breaks down the input prompt into a structured representation of sub-concepts and their relationships (e.g., "hedgehog holding rolling pin," "rolling pin rolling dough"). This hierarchical decomposition provides a detailed "checklist" for subsequent stages.
  2. MLLM-Based Critique and Refinement: An initial image is generated. Then, an MLLM (specifically GPT-40) critiques the image based on the decomposed concepts, identifying discrepancies and suggesting specific corrections. This critique is used to refine the original prompt.
  3. Refinement by Diffusion Re-denoising: The refined prompt is used to guide a partial re-denoising process of the initial image. By adding controlled noise and re-running the diffusion process, DetailScribe selectively corrects errors while preserving the overall image structure, leading to a more faithful and realistic output.

Datasets

  • InterActing Dataset: A newly introduced dataset of 1000 fine-grained, interaction-focused text-to-image prompts. It covers three categories: functional/action-based interactions, multi-subject interactions, and compositional spatial relationships. This dataset is designed to specifically challenge and evaluate T2I models on complex entity interactions.
  • MS-COCO and ImageNet: Mentioned as standard benchmarks for evaluating image generation quality, though the paper focuses on its own dataset for interaction-specific evaluation.

Key Insights and Ideas

  • The Power of Decomposition: Explicitly decomposing complex prompts into a structured hierarchy of sub-concepts significantly improves an MLLM's ability to critique and refine generated images. This structured approach allows for more targeted error detection and correction.
  • Iterative Refinement is Crucial: A simple prompt rewrite is not enough. The "generate-then-refine" cycle, particularly with a partial re-denoising step, is essential for correcting subtle interaction errors without destroying the initial image's coherence.
  • MLLMs as Sophisticated Judges: MLLMs, when guided by concept decomposition, can act as powerful critics, identifying nuanced interaction failures that traditional metrics might miss.
  • Interaction is the Next Frontier: While T2I models are good at objects, capturing the "how" and "why" of their interactions is the next major challenge, requiring specialized datasets and methods like DetailScribe.

Key Results

  • Superior Performance: DetailScribe consistently outperformed baselines (Stable Diffusion, SD + GPT Rewrite, SD + GPT Refine, SD (Alt. Seed), DALL-E 3) across all three scenarios (Functional Relation, Multi-subject Interaction, Compositional Relation) in both human and MLLM evaluations.
  • Human Preference: DetailScribe received the highest average human Likert scale scores, indicating a strong preference for its generated images.
  • MLLM Agreement: The MLLM evaluator (Qwen2.5-VL-32B) showed high agreement (87.6%) with human evaluations, validating its utility as an automatic metric.
  • Qualitative Improvements: Figures and examples demonstrate DetailScribe's ability to accurately depict complex interactions like a cat holding a mast, ants lifting a crumb, or a rabbit sculpting snow, which baselines often failed to capture correctly.

Limitations and Future Work

  • Reliance on Initial Generation: DetailScribe assumes the initial image has a roughly correct global scene structure. It struggles with large-scale edits like adding missing main subjects or significantly altering the overall layout, as the re-denoising step is not designed for drastic changes.
  • Global Scene Edits: The framework's effectiveness is limited when the initial generation is fundamentally flawed in its global structure.
  • Future Work: The authors suggest exploring seed search strategies (similar to Ma et al., 2025) to find better initial images and potentially sampling multiple initial images simultaneously with MLLM feedback for post-hoc selection to improve global scene accuracy. They also mention the possibility of extending the framework to multi-round refinements.

Summary updated

Abstract

Recent text-to-image models excel at generating high-quality object-centric images from instructions. However, images should also encapsulate rich interactions between objects, where existing models often fall short, likely due to limited training data and benchmarks for rare interactions. This paper explores a novel application of Multimodal Large Language Models (MLLMs) to benchmark and enhance the generation of interaction-rich images. We introduce \data, an interaction-focused dataset with 1000 LLM-generated fine-grained prompts for image generation covering (1) functional and action-based interactions, (2) multi-subject interactions, and (3) compositional spatial relationships. To address interaction-rich generation challenges, we propose a decomposition-augmented refinement procedure. Our approach, \model, leverages LLMs to decompose interactions into finer-grained concepts, uses an MLLM to critique generated images, and applies targeted refinements with a partial diffusion denoising process. Automatic and human evaluations show significantly improved image quality, demonstrating the potential of enhanced inference strategies.
BibTeX
@inproceedings{gu2025generating,
  author = {Xinyi Gu and Jiayuan Mao},
  title = {{Generating Fine Details of Entity Interactions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure