SHYI: Action Support for Contrastive Learning in High-Fidelity Text-to-Image Generation

Tianxiang Xia, Lin Xiao, Yannick Montorfani, Francesco Pavia, Enis Simsar, Thomas HofmannarXiv 2025 · DOI · arXiv

Abstract

Modern text-to-image diffusion models often struggle with "interaction infidelity," failing to accurately depict relationships between multiple objects in a scene. We present SHYI (Semantically Hypergraphic contrastive adjacency learning with InteractDiffusion enhancement), a novel framework that improves action-object alignment. By representing prompts as semantic hypergraphs and introducing a specialized "link loss" within a contrastive learning objective, SHYI ensures interacting objects maintain distinct identities while achieving the physical proximity required for realistic interaction. Evaluation via TIFA, CLIP scores, and human studies demonstrates that SHYI significantly outperforms existing baselines in generating complex multi-object scenes.

Introduction

While contemporary text-to-image models like Stable Diffusion produce high-quality individual objects, they frequently stumble when tasked with depicting interactions. A request for a "cat chasing a mouse" might yield two separate, well-rendered animals with no clear interaction, or a distorted hybrid. These failures stem from attribute binding errors and a fundamental lack of spatial-semantic coordination during the diffusion process.

Existing contrastive learning methods, such as CONFORM, focus on keeping object attention regions separate to prevent pixel "bleeding." However, this separation can inadvertently hinder interactions by pushing objects too far apart. SHYI addresses this gap by proposing a method to keep objects distinct while simultaneously "linking" them through the actions they perform.

Methodology

The SHYI framework employs a multi-pronged approach to spatial and semantic control:

1. Semantic Hypergraphs

The model interprets prompts as hypergraphs where nouns serve as vertices and actions act as "hyperedges" connecting them. For instance, in the prompt "A pigeon carrying a letter," the action carrying is an edge connecting the pigeon and the letter nodes.

Semantic hypergraph representation of a complex prompt where nouns are vertices and actions are hyperedges.

2. Contrastive Adjacency Learning

To balance object separation with interaction proximity, SHYI utilizes two primary loss functions:

  • Contrast Loss: Prevents the attention regions of different objects from overlapping, maintaining clear boundaries.
  • Link Loss: A novel addition ensuring that the borders of interacting objects remain close, facilitating the physical contact or proximity required for the specified action.
Contrastive Adjacency Learning: Using contrast loss for separation and link loss for interaction proximity, visualized through attention maps.

3. Optimization Strategies

To prevent actions from distorting object geometry, SHYI utilizes a "detached tensor" trick, where gradients for objects are detached during the action-learning phase. This forces the action to adapt to the objects rather than warping the objects to fit a vague action. Furthermore, an iterative loss selection process alternates between different loss terms at each gradient descent step to avoid conflicting signals between separation and connection.

Results and Evaluation

The researchers evaluated SHYI using a custom test suite of 14 prompts generated by ChatGPT-4-turbo, designed to test complex object-action relationships.

Quantitative and Qualitative Performance

  • User Preference: In a study with 30 participants, SHYI achieved an average win rate of 54.26% over the CONFORM baseline. For specific prompts like "Young girl reading a thick book," preference reached 79.33%.
  • Action Alignment: Qualitative results showed SHYI successfully generated interactions like "Fisherman catching fish," where baselines typically failed to establish a physical connection.
  • Metric Insights: The study revealed that standard metrics like TIFA and CLIP often fail to capture the nuances of actions (e.g., distinguishing "pulling" from "pushing"), underscoring the importance of human evaluation in this domain.

Conclusion and Future Work

SHYI represents a significant step toward high-fidelity interaction in text-to-image generation. By treating actions as semantic links, it bridges the gap between object separation and physical interaction. Current limitations include a "complexity cap" for scenes with more than two interacting objects and sensitivity to link loss hyperparameters. Future research will focus on automating spatial relationship priors to further refine the model's understanding of complex, multi-object environments.

Summary updated

Abstract

In this project, we address the issue of infidelity in text-to-image generation, particularly for actions involving multiple objects. For this we build on top of the CONFORM framework which uses Contrastive Learning to improve the accuracy of the generated image for multiple objects. However the depiction of actions which involves multiple different object has still large room for improvement. To improve, we employ semantically hypergraphic contrastive adjacency learning, a comprehension of enhanced contrastive structure and "contrast but link" technique. We further amend Stable Diffusion's understanding of actions by InteractDiffusion. As evaluation metrics we use image-text similarity CLIP and TIFA. In addition, we conducted a user study. Our method shows promising results even with verbs that Stable Diffusion understands mediocrely. We then provide future directions by analyzing the results. Our codebase can be found on polybox under the link: https://polybox.ethz.ch/index.php/s/dJm3SWyRohUrFxn
BibTeX
@misc{xia2025shyi,
  author = {Tianxiang Xia and Lin Xiao and Yannick Montorfani and Francesco Pavia and Enis Simsar and Thomas Hofmann},
  title = {{SHYI: Action Support for Contrastive Learning in High-Fidelity Text-to-Image Generation}},
  year = {2025},
  eprint = {2501.09055},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.09055},
}
Figure