Text-to-Image Generation Grounded by Fine-Grained User Attention

Jing Yu Koh, Jason Baldridge, Honglak Lee, Yinfei Yang

Abstract

Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TRECS, a sequential model that exploits this grounding to generate images. TRECS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.

Introduction: The Challenge of Spatial Control

Most text-to-image models, such as AttnGAN, attempt to generate pixels directly from short, simple captions. This approach faces two major hurdles: complexity and spatial control. Standard models often struggle with long, detailed descriptions, producing blurry "visual gists" rather than coherent objects. Furthermore, text alone is a poor medium for describing exact sizes and positions; a user might specify that a dog is on the left, but they lack fine-grained control over its specific pose or scale.

Comparison of standard captions versus localized narratives with mouse traces.

To address these gaps, the authors leverage the Localized Narratives dataset. In this dataset, annotators describe an image while simultaneously moving their mouse over the objects they mention, providing a rich, fine-grained signal of user attention that TRECS uses to ground the generation process.

The TRECS Framework

TRECS (Tag-Retrieve-Compose-Synthesize) is a four-stage pipeline that moves away from end-to-end "hallucination" toward a structured, retrieval-based approach:

  1. Tagging (Sequence Labeling): The system predicts which object labels correspond to specific words in the narrative. To handle noise in human mouse traces, a Hidden Markov Model (HMM) is used to clean the alignments, which then fine-tunes a BERT model for final tagging.
  2. Retrieval: Instead of drawing objects from scratch, TRECS uses a cross-modal dual encoder to search the COCO-Stuff database for existing segmentation masks that match the narrative's context (e.g., retrieving a "skier" mask rather than a generic "person").
  3. Composition: Retrieved masks are placed onto a canvas, selecting those that best overlap with the user's mouse traces. The system ensures a coherent, fully covered segmentation map by treating discrete objects ("things") and backgrounds ("stuff") differently.
  4. Synthesis: Finally, the composed segmentation map is passed to a pre-trained mask-to-image generator (such as SPADE) to produce a photo-realistic image.
The TRECS workflow: from user-provided descriptions and traces to a synthesized image.

Key Results and Insights

TRECS demonstrates that retrieval is an effective shortcut to realism. Generating a complex object like a skier in a specific pose is difficult for a GAN, but retrieving and "re-skinning" a real mask is much more effective. Mouse traces act as a spatial anchor, ensuring objects are placed exactly where intended.

In evaluations on the LN-COCO dataset, human raters preferred TRECS images 77% of the time over AttnGAN for realism. TRECS also achieved a lower (better) FID score of 48.7 compared to AttnGAN’s 51.8. Qualitatively, the model produces much sharper object boundaries and more recognizable entities compared to the "blobby" outputs of direct text-to-image GANs.

Limitations and Future Work

The quality of TRECS is currently limited by the diversity of its mask database; the model cannot "invent" a pose or object not already present. Additionally, noisy traces can occasionally lead to awkward spatial arrangements. Future research may explore incorporating fine-grained traces directly into end-to-end architectures to combine the strengths of both approaches.

Summary updated

Abstract

Localized Narratives [28] is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TRECS, a sequential model that exploits this grounding to generate images. TRECS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.
BibTeX
@inproceedings{koh2021text,
  author = {Jing Yu Koh and Jason Baldridge and Honglak Lee and Yinfei Yang},
  title = {{Text-to-Image Generation Grounded by Fine-Grained User Attention}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=l4WRQ-gxNB},
}
Figure