Abstract
Localized Narratives is a dataset with detailed natural language descriptions of images paired with mouse traces that provide a sparse, fine-grained visual grounding for phrases. We propose TReCS, a sequential model that exploits this grounding to generate images. TReCS uses descriptions to retrieve segmentation masks and predict object labels aligned with mouse traces. These alignments are used to select and position masks to generate a fully covered segmentation canvas; the final image is produced by a segmentation-to-image generator using this canvas. This multi-step, retrieval-based approach outperforms existing direct text-to-image generation models on both automatic metrics and human evaluations: overall, its generated images are more photo-realistic and better match descriptions.
Introduction
Traditional text-to-image (T2I) models, like AttnGAN, typically rely on short, 10-word captions (e.g., "a bird perched on a branch"). While these models can capture the general "gist" of a scene, they struggle with complex, detailed descriptions and often fail to place objects exactly where a user wants them.
The challenge lies in the "Localized Narratives" dataset, where descriptions average 42 words and are accompanied by mouse traces that show exactly where the speaker was looking while describing the scene. Existing end-to-end models aren't designed to ingest this high-resolution spatial intent. The authors address the gap between free-form natural language and precise spatial grounding, moving beyond "blobs" of color toward coherent, well-defined object composition.
Methods: The TRECS Pipeline
Instead of trying to "hallucinate" an image directly from text, the authors propose a four-stage pipeline called TRECS (Tag-Retrieve-Compose-Synthesize):
- Tagging: A BERT-based model predicts object labels for every word in the narrative. To train this without manual word-level annotations, the authors used a "constrained Hidden Markov Model (HMM)" to automatically align noisy mouse traces with semantic labels from the COCO-Stuff dataset.
- Retrieval: Using a cross-modal dual encoder (Inception v3 for images, BERT for text), the system retrieves candidate masks from a database that match the narrative's context. For example, if the text says "person skiing," it retrieves masks of skiers rather than generic "person" shapes.
- Composition: The system creates a segmentation canvas. "Thing" objects (like people or cars) are centered on the user's mouse traces. "Stuff" (backgrounds like sky or grass) is pulled from a single coherent background image to ensure the environment looks natural.
- Synthesis: Finally, the completed segmentation canvas is passed through a mask-to-image generator (specifically CC-FPSE), which converts the labels into a photo-realistic image.
Key Insights and Results
One of the primary findings is that for complex scenes, retrieving and "stitching" existing high-quality semantic masks is more effective than trying to generate them from scratch. This avoids the blurry, unidentifiable shapes often seen in end-to-end GAN outputs. Mouse traces provide a "sparse, fine-grained visual grounding." By using the convex hull of these traces, the model knows exactly the scale and position intended by the user, solving the common T2I problem of objects being the wrong size or in the wrong place.
TRECS demonstrated a clear lead over the previous state-of-the-art (AttnGAN):
- Human Preference: In side-by-side evaluations on LN-COCO, human raters preferred TRECS images for realism 77% of the time and for image-text alignment 52.4% of the time.
- Quantitative Metrics: On the LN-COCO dataset, TRECS achieved an Inception Score (IS) of 21.3 and a FID of 48.7, both significant improvements over AttnGAN (IS: 20.8, FID: 51.8).
- Generalization: Even on the more difficult OpenImages dataset, TRECS maintained a strong lead in human-judged realism (77.2% preference).
Conclusion
While TRECS relies on a database of masks, which limits it to known object categories, its retrieval-based logic provides a powerful alternative to end-to-end hallucination. Future work may involve integrating this logic into a unified end-to-end framework or expanding the system to support interactive, iterative editing where users add traces and text sequentially.
Summary updated