Object-centric Self-improving Preference Optimization for Text-to-Image Generation

Yoonjin Oh, Yongjin Kim, Hyomin Kim, Donghwan Chi, Sungwoong KimarXiv 2025 · DOI · arXiv

Introduction

The emergence of Unified Multimodal Large Language Models (MLLMs), such as Janus and Emu, has marked a significant shift toward models capable of both understanding and generating visual content. Despite their versatility, these models frequently suffer from "object hallucinations"—instances where they fail to faithfully render specific attributes, spatial relations, or quantities described in a text prompt. For example, a request for "a navy helmet and green apples" might result in a green helmet or missing fruit.

While Preference Optimization techniques like Direct Preference Optimization (DPO) have been used to align model outputs with human expectations, they rely on massive, expensive human-labeled datasets. Existing self-improvement frameworks attempt to bypass this by having the model critique its own outputs, but these often operate at a "coarse" image-wide level, failing to address the fine-grained, object-specific errors that degrade text-to-image alignment.

The OSPO Framework

To address these challenges, we introduce Object-centric Self-improving Preference Optimization (OSPO). OSPO is a five-stage pipeline that enables an MLLM to serve as its own teacher, identifying and correcting its own generative flaws without human intervention.

The five-stage OSPO pipeline: from prompt perturbation to object-weighted preference optimization.

1. Prompt Generation and Perturbation

The process begins by generating a base prompt. To create a clear learning signal, the model generates "perturbed" versions of this prompt. By swapping attributes (e.g., changing "red car" to "blue car") or replacing objects, the system creates pairs of prompts that are nearly identical except for one specific detail.

2. Prompt Densification

To ensure the model learns to render objects within a realistic environment rather than in isolation, OSPO "densifies" the prompts. It adds descriptive context—such as "a sturdy navy helmet... on a dusty bench"—to provide a consistent global background for both the original and perturbed prompts.

3. Image and Object Mask Generation

The model generates images for the prompt pairs. A key innovation of OSPO is its use of internal attention masks. Instead of relying on external segmentation models like SAM or Grounding DINO, OSPO extracts object masks directly from the model’s internal attention weights. By observing which visual tokens the model attends to when processing an object's name, the system identifies the exact pixels corresponding to that object.

4. VQA-based Self-Evaluation

OSPO leverages the inherent asymmetry in MLLMs: they are typically better at understanding images than generating them. The model acts as a judge by asking itself a series of decompositional Yes/No questions (e.g., "Is the helmet navy?"). These answers are used to score the generated images and construct preference pairs.

5. Object-weighted SimPO

Finally, the model is fine-tuned using a modified version of Simple Preference Optimization (SimPO). By using the masks generated in Stage 3, OSPO applies an object-weighted loss. This forces the model to prioritize the specific pixels representing the objects it failed to render correctly, leading to much sharper improvements in compositional fidelity.

Experimental Setup and Results

OSPO was trained on a self-generated dataset of 20,000 prompts across four categories: Attribute, Layout, Non-spatial Relationship, and Complex Composition. Evaluation was conducted using rigorous benchmarks including T2I-CompBench++, DPGBench, and GenEval.

Scaling analysis of OSPO: Performance improves consistently with training data size and the number of candidate image pairs (N).

The results demonstrate that OSPO significantly enhances the generative capabilities of MLLMs. For instance, the Janus-Pro 7B model saw its "Color" attribute score on T2I-CompBench++ jump from 0.52 to 0.85. Furthermore, OSPO-trained models frequently outperformed specialized diffusion models like SD-XL and DALL-E 3 in attribute rendering and spatial layout accuracy.

Key Insights

  • Exploiting the Understanding-Generation Gap: By using the model's superior vision capabilities to supervise its drawing skills, OSPO creates a virtuous cycle of self-improvement.
  • Efficiency through Precision: Unlike methods that require sampling dozens of images to find a "good" one, OSPO achieves high performance with very small candidate sets (N=2 or 3), making it computationally efficient.
  • Attention as Supervision: Utilizing internal attention maps provides a "free" and accurate source of object-level supervision, avoiding the overhead of external models.

Limitations and Future Work

While OSPO marks a major step forward, challenges remain. The model still struggles with "Numeracy" (e.g., drawing the exact number of requested items). Additionally, the system's improvement is bounded by its pre-existing internal knowledge; it cannot teach itself to draw objects it has never encountered during pre-training. Future research will explore iterative training rounds and the application of object-centric optimization to video generation.

Summary updated

BibTeX
@misc{oh2025object,
  author = {Yoonjin Oh and Yongjin Kim and Hyomin Kim and Donghwan Chi and Sungwoong Kim},
  title = {{Object-centric Self-improving Preference Optimization for Text-to-Image Generation}},
  year = {2025},
  eprint = {2506.02015},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.02015},
}
Figure