Abstract
Generating complex scenes with multiple specific objects remains a significant challenge for text-to-image models. Current systems often suffer from "attribute leakage," where properties like color bleed between objects, and "relationship discrepancy," where objects fail to interact as described. RaDL (Relation-aware Disentangled Learning) addresses these issues through a novel framework that combines instance-specific attribute boosting with a "Relation Attention" mechanism focused on action verbs. RaDL ensures that objects maintain their distinct identities while behaving realistically within the scene.
Introduction
While generating a single object from text is largely considered a solved problem, creating scenes with multiple interacting objects—such as "a blue surfboard leaning against a white table"—is surprisingly difficult. Existing Layout-to-Image (L2I) models frequently fail in two ways:
- Attribute Leakage: Models often confuse attributes, applying the surfboard's color to the table because text features are processed in a shared space.
- Relationship Discrepancy: Even when objects are placed correctly, they often ignore interaction instructions. A surfboard might be near a table but not actually "leaning against" it.
RaDL overcomes these hurdles by maintaining object individuality while enforcing global relationships through a structured three-stage pipeline.
Methods
RaDL introduces a three-stage architecture designed to preserve object integrity while modeling complex interactions:
- Multi-Instance Disentanglement: The framework uses instance masks to isolate the features of each object, ensuring that attributes like "blue" remain anchored to the "surfboard" and do not drift toward other instances.
- Instance-Specific Modular Attention: This module employs learnable parameters to enhance specific attributes. It anchors textual descriptions (e.g., "silver metal") to the specific image region defined by a bounding box, preventing visual degradation during generation.
- Multi-Stage Semantic Instance Fusion (Relation Attention): This core innovation extracts action verbs (e.g., "leaning," "riding") from the prompt. A "Relation Attention" layer then applies these verbs across a total instance mask, forcing the model to consider how different objects should blend or interact at their boundaries.
Key Results and Evaluation
RaDL was evaluated against major benchmarks including COCO-Position, COCO-MIG (Multi-Instance Generation), and DrawBench. The model consistently outperformed state-of-the-art baselines like GLIGEN and MIGC across all key metrics:
- Relationship Accuracy: On DrawBench, RaDL achieved a 73.54% success rate in representing relationships, a significant improvement over MIGC’s 60.83%.
- Attribute Preservation: In COCO-MIG tests involving up to six instances, RaDL maintained a higher Mean IoU (53.27) and Success Rate (59.45%) than competitors, proving its effectiveness in preventing attribute leakage.
- Spatial Precision: On COCO-Position, RaDL reached a Success Rate of 81.09%, surpassing previous benchmarks while maintaining high image quality (FID score of 23.53).
Conclusion and Future Work
RaDL demonstrates that verbs are the essential "glue" for realistic multi-object scenes. By explicitly encoding action-based verbs and using learnable attribute anchors, the model achieves a new standard for complex scene generation. While currently dependent on user-provided bounding boxes, future iterations may move toward segmentation masks or keypoints to allow for even more precise interaction modeling, such as specific hand-object grasping.
Summary updated