Generating Fine Details of Entity Interactions

Xinyi Gu, Jiayuan Mao

While modern text-to-image (T2I) models produce high-quality individual objects, they frequently fail to accurately depict complex, fine-grained interactions between entities. This paper introduces InterActing, a new benchmark specifically designed to evaluate these "interaction-rich" scenarios. To address the identified shortcomings, we propose DetailScribe, a "generate-then-refine" framework. DetailScribe leverages Multimodal Large Language Models (MLLMs) to decompose prompts into structured visual checklists, critique initial generations for semantic inaccuracies, and perform targeted refinements through partial re-denoising.

Introduction

The rapid advancement of diffusion models has enabled the generation of visually stunning images from natural language prompts. However, a significant challenge remains: the precise rendering of intricate entity interactions. Models often struggle with "semantic glitches"—scenarios where physical or functional relationships between objects are distorted or ignored. For instance, a model might generate a beaver and a pizza but fail to show the beaver actually cutting the pizza with a tool. These failures often stem from the underrepresentation of such specific, high-detail interactions in large-scale training datasets.

The InterActing Benchmark

To systematically study these failures, we introduced the InterActing Dataset. This benchmark consists of 1,000 prompts categorized into three challenging interaction types:

  1. Functional and Action-Based Interactions: Tasks involving tool use (e.g., painting, cutting) or complex physical contact (e.g., sculpting).
  2. Multi-subject Interactions: Collaborative or social actions between multiple entities, such as sharing food or performing a high-five.
  3. Compositional Spatial Relationships: Precise geometric or abstract layouts, such as a "zig-zag" path of leaves or specific arrangements of objects.

The DetailScribe Framework

DetailScribe addresses interaction failures through a three-stage refinement process that moves beyond simple prompt engineering.

The DetailScribe pipeline: (a) Task Input, (b) Concept Decomposition via LLM, (c) Self-Critique and Correction using an MLLM, leading to a refined Round 1 generation.
  1. Concept Decomposition: An LLM (e.g., GPT-4o) transforms a prompt into a "visual abstraction schema." This schema breaks down the main action into verifiable sub-concepts. For a prompt like "a hedgehog rolling dough," the schema includes requirements such as "paws grasping a rolling pin" and "dough spread out on the table."
  2. MLLM-Based Critique: A base T2I model generates an initial image. An MLLM then evaluates this image against the decomposed checklist. By focusing on specific interaction points rather than global image quality, the MLLM can pinpoint exact failures (e.g., a hand hovering near, but not touching, an object).
  3. Partial Re-denoising: Rather than a full regeneration, which might discard successful parts of the image, DetailScribe uses a "partial re-denoising" strategy. It adds a controlled amount of noise back into the latent representation (typically at step $T-2$ of the diffusion process) and re-runs the denoising steps using a refined, detail-oriented prompt provided by the MLLM. This preserves the overall composition while fixing the specific interaction errors.

Experimental Results

DetailScribe was evaluated against leading models including Stable Diffusion 3.5 (SD3.5) and DALL-E 3 across the InterActing benchmark.

Qualitative comparison between Stable Diffusion and DetailScribe. Red circles highlight interaction failures in the base model, while green circles show the successful refinements made by DetailScribe.

Quantitative Performance

Human evaluation on a 1-5 Likert scale for image-text alignment showed that DetailScribe consistently outperformed baselines. In "Functional Relations," DetailScribe achieved a score of 4.28, significantly higher than SD3.5 (3.36) and DALL-E 3 (3.94). Automated metrics, including ImageReward and BLIP-VQA, further confirmed these improvements, with MLLM-based evaluations showing an 87.6% agreement rate with human judges.

Qualitative Success

The framework proved particularly effective at capturing nuanced details that other models often overlook. Examples include a "cat sailing by holding a mast" or "autumn leaves in a zig-zag pattern." In these cases, DetailScribe's iterative refinement ensured that the specific verbs and spatial constraints were respected, whereas base models often ignored the "holding" or "zig-zag" descriptors.

Conclusion and Future Work

DetailScribe demonstrates that MLLMs can serve as effective "critics" to guide diffusion models toward better semantic alignment. By decomposing complex prompts into manageable checklists and using partial re-denoising, we can fix fine-grained interaction errors without losing global image structure. Future work will explore multi-round refinements and "seed search" strategies to further enhance the robustness of the generation process.

Summary updated

BibTeX
@inproceedings{gu2025generating,
  author = {Xinyi Gu and Jiayuan Mao},
  title = {{Generating Fine Details of Entity Interactions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure