RaDL: Relation-aware Disentangled Learning for Multi-Instance Text-to-Image Generation

Geon Park, Seon Bin Kim, Gunho Jung, Seong-Whan LeearXiv 2025 · DOI · arXiv

Abstract

Generating complex scenes with multiple specific objects remains a significant challenge for text-to-image models. Current systems often suffer from "attribute leakage," where properties like color bleed between objects, and "relationship discrepancy," where objects fail to interact as described. RaDL (Relation-aware Disentangled Learning) addresses these issues through a novel framework that combines instance-specific attribute boosting with a "Relation Attention" mechanism focused on action verbs. RaDL ensures that objects maintain their distinct identities while behaving realistically within the scene.

Introduction

While generating a single object from text is largely considered a solved problem, creating scenes with multiple interacting objects—such as "a blue surfboard leaning against a white table"—is surprisingly difficult. Existing Layout-to-Image (L2I) models frequently fail in two ways:

  1. Attribute Leakage: Models often confuse attributes, applying the surfboard's color to the table because text features are processed in a shared space.
  2. Relationship Discrepancy: Even when objects are placed correctly, they often ignore interaction instructions. A surfboard might be near a table but not actually "leaning against" it.

RaDL overcomes these hurdles by maintaining object individuality while enforcing global relationships through a structured three-stage pipeline.

Methods

RaDL introduces a three-stage architecture designed to preserve object integrity while modeling complex interactions:

  • Multi-Instance Disentanglement: The framework uses instance masks to isolate the features of each object, ensuring that attributes like "blue" remain anchored to the "surfboard" and do not drift toward other instances.
  • Instance-Specific Modular Attention: This module employs learnable parameters to enhance specific attributes. It anchors textual descriptions (e.g., "silver metal") to the specific image region defined by a bounding box, preventing visual degradation during generation.
  • Multi-Stage Semantic Instance Fusion (Relation Attention): This core innovation extracts action verbs (e.g., "leaning," "riding") from the prompt. A "Relation Attention" layer then applies these verbs across a total instance mask, forcing the model to consider how different objects should blend or interact at their boundaries.
The RaDL architecture featuring Multi-Instance Disentanglement, Instance-Specific Modular Attention, and Semantic Instance Fusion modules.

Key Results and Evaluation

RaDL was evaluated against major benchmarks including COCO-Position, COCO-MIG (Multi-Instance Generation), and DrawBench. The model consistently outperformed state-of-the-art baselines like GLIGEN and MIGC across all key metrics:

  • Relationship Accuracy: On DrawBench, RaDL achieved a 73.54% success rate in representing relationships, a significant improvement over MIGC’s 60.83%.
  • Attribute Preservation: In COCO-MIG tests involving up to six instances, RaDL maintained a higher Mean IoU (53.27) and Success Rate (59.45%) than competitors, proving its effectiveness in preventing attribute leakage.
  • Spatial Precision: On COCO-Position, RaDL reached a Success Rate of 81.09%, surpassing previous benchmarks while maintaining high image quality (FID score of 23.53).
Qualitative comparison of RaDL against several baseline models, demonstrating superior attribute preservation and layout adherence.

Conclusion and Future Work

RaDL demonstrates that verbs are the essential "glue" for realistic multi-object scenes. By explicitly encoding action-based verbs and using learnable attribute anchors, the model achieves a new standard for complex scene generation. While currently dependent on user-provided bounding boxes, future iterations may move toward segmentation masks or keypoints to allow for even more precise interaction modeling, such as specific hand-object grasping.

Summary updated

BibTeX
@misc{park2025radl,
  author = {Geon Park and Seon Bin Kim and Gunho Jung and Seong-Whan Lee},
  title = {{RaDL: Relation-aware Disentangled Learning for Multi-Instance Text-to-Image Generation}},
  year = {2025},
  eprint = {2507.11947},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.11947},
}
Figure