Abstract
Modern text-to-image diffusion models often struggle with "interaction infidelity," failing to accurately depict relationships between multiple objects in a scene. We present SHYI (Semantically Hypergraphic contrastive adjacency learning with InteractDiffusion enhancement), a novel framework that improves action-object alignment. By representing prompts as semantic hypergraphs and introducing a specialized "link loss" within a contrastive learning objective, SHYI ensures interacting objects maintain distinct identities while achieving the physical proximity required for realistic interaction. Evaluation via TIFA, CLIP scores, and human studies demonstrates that SHYI significantly outperforms existing baselines in generating complex multi-object scenes.
Introduction
While contemporary text-to-image models like Stable Diffusion produce high-quality individual objects, they frequently stumble when tasked with depicting interactions. A request for a "cat chasing a mouse" might yield two separate, well-rendered animals with no clear interaction, or a distorted hybrid. These failures stem from attribute binding errors and a fundamental lack of spatial-semantic coordination during the diffusion process.
Existing contrastive learning methods, such as CONFORM, focus on keeping object attention regions separate to prevent pixel "bleeding." However, this separation can inadvertently hinder interactions by pushing objects too far apart. SHYI addresses this gap by proposing a method to keep objects distinct while simultaneously "linking" them through the actions they perform.
Methodology
The SHYI framework employs a multi-pronged approach to spatial and semantic control:
1. Semantic Hypergraphs
The model interprets prompts as hypergraphs where nouns serve as vertices and actions act as "hyperedges" connecting them. For instance, in the prompt "A pigeon carrying a letter," the action carrying is an edge connecting the pigeon and the letter nodes.
2. Contrastive Adjacency Learning
To balance object separation with interaction proximity, SHYI utilizes two primary loss functions:
- Contrast Loss: Prevents the attention regions of different objects from overlapping, maintaining clear boundaries.
- Link Loss: A novel addition ensuring that the borders of interacting objects remain close, facilitating the physical contact or proximity required for the specified action.
3. Optimization Strategies
To prevent actions from distorting object geometry, SHYI utilizes a "detached tensor" trick, where gradients for objects are detached during the action-learning phase. This forces the action to adapt to the objects rather than warping the objects to fit a vague action. Furthermore, an iterative loss selection process alternates between different loss terms at each gradient descent step to avoid conflicting signals between separation and connection.
Results and Evaluation
The researchers evaluated SHYI using a custom test suite of 14 prompts generated by ChatGPT-4-turbo, designed to test complex object-action relationships.
Quantitative and Qualitative Performance
- User Preference: In a study with 30 participants, SHYI achieved an average win rate of 54.26% over the CONFORM baseline. For specific prompts like "Young girl reading a thick book," preference reached 79.33%.
- Action Alignment: Qualitative results showed SHYI successfully generated interactions like "Fisherman catching fish," where baselines typically failed to establish a physical connection.
- Metric Insights: The study revealed that standard metrics like TIFA and CLIP often fail to capture the nuances of actions (e.g., distinguishing "pulling" from "pushing"), underscoring the importance of human evaluation in this domain.
Conclusion and Future Work
SHYI represents a significant step toward high-fidelity interaction in text-to-image generation. By treating actions as semantic links, it bridges the gap between object separation and physical interaction. Current limitations include a "complexity cap" for scenes with more than two interacting objects and sensitivity to link loss hyperparameters. Future research will focus on automating spatial relationship priors to further refine the model's understanding of complex, multi-object environments.
Summary updated