O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model

Rishi Gupta, Mukilan Karuppasamy, Shyam Marjit, Aditay Tripathi, Anirban Chakraborty

TL;DR

While Large Vision-Language Models (LVLMs) excel at processing text and natural images, they remain surprisingly blind to abstract, hand-drawn sketches. This paper introduces SketchVCL, a massive multi-task dataset of image-sketch-instruction triplets, and O3SLM, the first open-weight, open-vocabulary Sketch-Language Model. O3SLM achieves state-of-the-art performance in sketch comprehension, counting, and localization, significantly outperforming closed-source giants like GPT-4o.

What Problem Does This Paper Address?

Hand-drawn sketches are an intuitive, universal way to communicate complex visual ideas or specific object attributes that are difficult to describe with text alone. However, current state-of-the-art LVLMs struggle to understand them. As shown in Figure 1, models like LLaVA and GPT-4o fail at downstream tasks like counting or identifying objects from a crude sketch query because they treat the sketch as an abstract geometric shape rather than a semantic prompt.

Figure 1: Limitations of LVLMs in Sketch Understand- ing. Although current LVLMs can interpret sketches to some level of abstraction, they struggle in sketch understanding for downstream tasks like detection and reasoning.

The authors identify the core bottleneck: the lack of a large-scale, diverse dataset that jointly models sketches, photorealistic images, and natural language instructions. Existing sketch datasets are either class-wise only (lacking paired images) or limited to simple, single-object scenes without textual annotations.

Methods

To bridge this gap, the authors developed a unified framework consisting of an automated data curation pipeline and a two-stage training strategy:

  1. Automated Sketch Generation Pipeline: To scale up data generation, the authors used Segment Anything 2 (SAM2) to mask object backgrounds from real images. They then passed the foreground through a Pix2Pix model (Photo2Sketch) to generate abstract sketches, which were further enhanced using morphological gradient edge detection.
  2. O3SLM Architecture: Built on top of LLaVA-1.5, O3SLM uses a CLIP ViT-L/336 visual backbone to encode both sketches and natural images, a 2-layer MLP connector, and a Vicuna-v1.5 LLM base. Sketch, image, and text tokens are concatenated and fed directly into the LLM, leveraging its internal self-attention mechanism.
  3. Two-Stage Training:
  4. Stage I: Sketch Alignment: Pretraining on 600k image-sketch pairs with detailed descriptive captions (generated by DeepSeek-VL2 and refined by LLaMA-3-8B Instruct) to teach the model 3-way alignment between text, sketches, and images.
  5. Stage II: Instruction Tuning: Fine-tuning across four downstream tasks (Counting, Object Localization, VQA, and SBIR) using task-specific natural language prefixes (e.g., COUNT, BBOX) to align the model's output format.

Datasets

  • SketchVCL (New): A large-scale dataset containing 600k pretraining instructions derived from Objects365 and OpenImages, and 215k instruction-tuning triplets based on COCO, Pixmo-Count, and Sketchy.
  • SketchMIX (New): A diverse sketch pool combining sketches from SketchVCL, QuickDraw!, and Sketchy, all mapped to the Objects365 class taxonomy to ensure broad visual diversity.
  • Evaluation Benchmarks: The model was evaluated on COCO, Pixmo-Count, Sketchy, QuickDraw!, and TU-Berlin (which was intentionally excluded from training to test zero-shot style generalization).

Key Insights and Ideas

  • Sketches as Semantic Queries: By aligning sketches directly with text and images during pretraining, O3SLM learns to treat a sketch as a semantic query rather than just a geometric pattern.
  • Emergent Fine-Grained Cross-Modal Understanding: Even though the model wasn't explicitly trained on combined sketch-text queries for every task, the multi-task curriculum (specifically VQA acting as an auxiliary signal) allows it to handle complex, fine-grained queries like "find the object depicted by the sketch in green or with a zebra print" (see Figure 2).
  • Unfreezing the Projector Matters: The authors found that tuning the multimodal projector alongside the LLM yields massive performance gains for sketch-image alignment. In fact, a 7B model with a tuned projector can outperform a 13B model with a frozen projector.
Capabilities of O3SLM across multiple sketch-based tasks.

Key Results

  • Sketch-Based Counting: On the challenging COCO-Count benchmark using Sketchy queries, O3SLM-13B achieved 36.7% accuracy, vastly outperforming LLaVA-1.5-13B (2.8%) and comfortably beating GPT-4o (13.9%).
  • Sketch-Based Image Retrieval (SBIR): On the Sketchy dataset, O3SLM-7B achieved an Acc@1 of 65.0%, compared to just 11.0% for the baseline LLaVA-1.5-7B.
  • Zero-Shot Style Generalization: When tested on the unseen TU-Berlin sketch dataset, O3SLM maintained top performance, proving robust cross-modal transfer to completely novel drawing styles.
  • Preserving Image-Only Capabilities: O3SLM retains its core vision-language capabilities, showing less than a 5% performance drop on standard image-only benchmarks (like VQAv2 and MME) while significantly improving text-only object detection.

Limitations and Future Work

The authors note that O3SLM struggles in crowded scenes containing a high density of overlapping objects, where it occasionally generates extra, redundant bounding boxes. Future work will explore integrating traditional detection algorithms like non-maximal suppression (NMS) directly into the pipeline or refining the model's spatial decoding capabilities to better handle cluttered environments.

Summary updated

Abstract

While Large Vision Language Models (LVLMs) are increasingly deployed in real-world applications, their ability to interpret abstract visual inputs remains limited. Specifically, they struggle to comprehend hand-drawn sketches, a modality that offers an intuitive means of expressing concepts that are difficult to describe textually. We identify the primary bottleneck as the absence of a large-scale dataset that jointly models sketches, photorealistic images, and corresponding natural language instructions. To address this, we present two key contributions: (1) a new, large-scale dataset of image-sketch-instruction triplets designed to facilitate both pretraining and instruction tuning, and (2) O3SLM, an LVLM trained on this dataset. Comprehensive evaluations on multiple sketch-based tasks: (a) object localization, (b) counting, (c) image retrieval i.e., (SBIR and fine-grained SBIR), and (d) visual question answering (VQA); while incorporating the three existing sketch datasets, namely QuickDraw!, Sketchy, and Tu Berlin, along with our generated SketchVCL dataset, show that O3SLM achieves state-of-the-art performance, substantially outperforming existing LVLMs in sketch comprehension and reasoning.
BibTeX
@inproceedings{gupta2026o,
  author = {Rishi Gupta and Mukilan Karuppasamy and Shyam Marjit and Aditay Tripathi and Anirban Chakraborty},
  title = {{O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i6.42450},
}
Figure