A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch

Patsorn Sangkloy, Wittawat Jitkrittum, Diyi Yang, James Hays

TL;DR

TASK-former (Text And SKetch transformer) is a novel dual-encoder model that retrieves images using a combination of text descriptions and hand-drawn sketches. By fusing these two complementary modalities, the model achieves a significant boost in retrieval accuracy—reaching 60.9% Recall@1 on COCO—outperforming state-of-the-art text-only methods even when the input sketches are crude or incomplete.

What Problem Does This Paper Address?

Traditional image retrieval usually falls into two camps: Text-Based Image Retrieval (TBIR) and Sketch-Based Image Retrieval (SBIR).

TBIR is excellent for describing attributes like color or category ("a red flower"), but it becomes incredibly cumbersome when you need to specify spatial layouts or complex shapes ("a fork to the left of a plate with meat and broccoli"). Conversely, SBIR is great for layout but difficult for non-artists to use for specific semantics; a quick doodle of a flower rarely conveys that it should be "wavy and red."

Furthermore, most SBIR research focuses on "single-object" retrieval (e.g., finding a specific shoe). This paper tackles in-the-wild retrieval, where scenes are complex, contain multiple objects, and the user's sketch is likely to be a rough "memory-based" drawing rather than a masterpiece.

Fig. 1: We present TASK-former, a Text And SKetch transformer based method for image retrieval. We demonstrate that the presence of a sketch input, even a poorly drawn one, helps narrow down the set of retrieved images to ones that match the joint description provided by the sketch and the text query. Retrieval results are from our model trained on Flower102 dataset [29]

Methods

The authors propose TASK-former, which adopts a late-fusion dual-encoder architecture inspired by CLIP. The system consists of three main encoders: one for text, one for sketches, and one for images.

Key Architectural Choices:

  • Shared Visual Backbone: The sketch and image encoders share weights (ViT-B/16), treating sketches as a specialized visual domain.
  • Simple Fusion: Surprisingly, the authors found that simply adding the text and sketch embeddings together performed better than more complex methods like concatenation or coordinate-wise maximums.
  • Training with Synthetic Data: To overcome the lack of massive sketch-text-image datasets, they trained the model using synthetically generated sketches derived from real images, applying heavy augmentations (affine transforms and stroke dropout) to mimic the messiness of human drawings.

The Triple-Loss Objective:

To ensure the learned embeddings are discriminative and semantically rich, they use three losses:

  1. Embedding Loss ($L_e$): A standard contrastive loss to align the joint (text+sketch) embedding with the target image embedding.
  2. Classification Loss ($L_c$): A multi-label loss that forces the encoders to recognize specific object categories.
  3. Caption Generation Loss ($L_d$): An auxiliary task where a GPT-style decoder tries to reconstruct the original text caption from the joint embedding. This ensures the embedding doesn't "forget" the fine-grained details provided in the text.
Fig. 2: Overview of our model and the training pipeline. ViT is the vision trans- former [12], and ⊕represents an element-wise addition. We extend CLIP [35] to incorporate additional sketch input from user. We add two auxiliary tasks: multi-label classification (Lc) and caption generation (Ld). Our objective losses are explained in Section 3.2. We encourages the network to learn a discrimina- tive representation to simultaneously distinguish at the instance level and at the category level, both of which are crucial for a successful retrieval.

Datasets

  • Training: The model was trained on the COCO dataset using synthetically generated sketches.
  • New Benchmark: The authors introduced a new evaluation dataset consisting of 5,000 hand-drawn sketches for the COCO test set. These were collected via Amazon Mechanical Turk, where participants drew scenes from memory after viewing an image for 15 seconds. This mimics real-world usage much better than sketches traced over images.

Key Insights and Ideas

  • Modality Synergy: Text and sketch are truly complementary. Text provides the "what" (semantics/colors), while sketch provides the "where" and "how" (spatial layout/shape).
  • Robustness to "Bad" Art: You don't need to be an artist. The researchers found that even a sketch with only 20% of its strokes intact provides a massive boost in retrieval recall compared to using text alone.
  • Weight Sharing Matters: Using the same ViT backbone for both images and sketches significantly improved performance, likely because it leverages the pre-trained visual features of CLIP more effectively.
  • The Power of Addition: The success of element-wise addition for fusion suggests that the model learns to map text and sketch features into a truly shared semantic space where their vectors can be meaningfully combined.

Key Results

The results demonstrate that adding a sketch—even a poor one—is a game-changer for retrieval:

  • Recall@1 on COCO: TASK-former achieved 60.9%, significantly higher than a fine-tuned text-only CLIP baseline (51.8%) and other SOTA models like ALIGN (59.9%) and OSCAR (57.5%).
  • Sketch Complexity: Reducing sketch completeness from 100% to 60% only resulted in a marginal drop in recall, proving the model's robustness to minimal user effort.
  • Zero-Shot Capability: The model can operate with only one modality (text-only or sketch-only) thanks to "query drop-out" during training, though performance is best when both are present.

Limitations and Future Work

  • Semantic Confusion: The model sometimes struggles with complex text descriptions involving multiple colors (e.g., confusing a "blue jacket" with a "black horse" if both colors appear in the prompt).
  • Spatial Sensitivity: While better than text-only, the model can still fail if there is a significant mismatch in scale or location between the user's sketch and the actual image.
  • Future Directions: The authors suggest exploring more sophisticated text augmentation and developing models that are more tolerant of scale mismatches to further bridge the gap between mental images and search results.

Summary updated

Abstract

We address the problem of retrieving images with both a sketch and a text query. We present TASK-former (Text And SKetch transformer), an end-to-end trainable model for image retrieval using a text description and a sketch as input. We argue that both input modalities complement each other in a manner that cannot be achieved easily by either one alone. TASK-former follows the late-fusion dual-encoder approach, similar to CLIP, which allows efficient and scalable retrieval since the retrieval set can be indexed independently of the queries. We empirically demonstrate that using an input sketch (even a poorly drawn one) in addition to text considerably increases retrieval recall compared to traditional text-based image retrieval. To evaluate our approach, we collect 5,000 hand-drawn sketches for images in the test set of the COCO dataset. The collected sketches are available a https://janesjanes.github.io/tsbir/.
BibTeX
@inproceedings{sangkloy2022sketch,
  author = {Patsorn Sangkloy and Wittawat Jitkrittum and Diyi Yang and James Hays},
  title = {{A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-19839-7\_15},
}
Figure