O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model

Rishi Gupta, Mukilan Karuppasamy, Shyam Marjit, Aditay Tripathi, Anirban Chakraborty

TL;DR

This paper introduces O3SLM, an open-weight, open-data model that significantly improves Large Vision-Language Models' (LVLMs) ability to understand hand-drawn sketches. They achieve this by releasing SketchVCL, a novel, large-scale dataset of image-sketch-instruction triplets, and demonstrating O3SLM's state-of-the-art performance on various sketch-based reasoning tasks.

What Problem Does This Paper Address?

Current Large Vision-Language Models (LVLMs) excel at understanding photorealistic images but struggle with more abstract visual inputs like hand-drawn sketches. This limitation hinders their application in scenarios where sketches are used for intuitive communication or as a more universal language. The core problem is the lack of a comprehensive, large-scale dataset that jointly models sketches, real images, and natural language instructions, which is crucial for training LVLMs to effectively interpret and reason about sketches.

Methods

The authors propose a two-stage training approach for their O3SLM model:

  1. Stage I: Sketch Alignment (Pretraining): This stage focuses on aligning sketches with images and text. They use a large-scale dataset (SketchVCL-OI, SketchVCL-O365) to teach the model to recognize sketches, associate them with objects in natural images, develop spatial understanding, and retain natural language capabilities. This ensures a "3-way alignment" of sketches, images, and text.
  2. Stage II: Instruction Tuning: This stage fine-tunes the model on specific sketch-based tasks. They train O3SLM on four tasks: Counting, Object Localization, Visual Question Answering (VQA), and Sketch-Based Image Retrieval (SBIR), using a curated task-specific dataset. They employ a prefix-based approach for task descriptors rather than modifying the vocabulary.

The O3SLM architecture itself uses a CLIP ViT-L/336 as the visual backbone, a multimodal connector (two-layered MLP) to project features into the LLM's input space, and the Vicuna v1.5 LLM as the base language model. They leverage LLaVA-1.5 weights for initialization.

Datasets

The paper introduces SketchVCL, a novel, large-scale dataset designed for sketch-image-text alignment and instruction tuning. SketchVCL is composed of image-sketch-instruction triplets and is built using an automated sketch generation pipeline. It includes:

  • SketchVCL-OI and SketchVCL-O365: For pretraining, derived from OpenImages and Object365 datasets, respectively.
  • SketchMIX: For instruction tuning, a mixture of sketches from various sources including SketchVCL-O365, SketchVCL-C, QuickDraw!, Sketchy, and TU-Berlin.

Existing datasets like QuickDraw!, Sketchy, and TU-Berlin are also incorporated into SketchMIX and used for evaluation.

Key Insights and Ideas

  • Sketches as a Crucial Modality: The paper strongly advocates for the importance of sketches as a visual modality for LVLMs, highlighting their expressive power and ability to transcend linguistic barriers.
  • The "Sketch Gap" in LVLMs: Existing LVLMs exhibit a significant performance drop when presented with sketches, indicating a clear gap in their understanding of abstract visual inputs.
  • Dataset is Key: The primary bottleneck identified is the lack of a suitable large-scale dataset. The creation of SketchVCL is presented as a direct solution to this problem.
  • Pretraining is Essential for Sketches: The results show that large-scale pretraining (Stage I) is crucial for achieving good performance on sketch-based tasks, particularly for SBIR, significantly boosting performance over models trained only with fine-tuning.
  • Emergent Fine-Grained Understanding: O3SLM demonstrates an emergent ability to perform fine-grained SBIR tasks, even when not explicitly trained for it, by leveraging the multitask pretraining and VQA supervision.

Key Results

  • State-of-the-Art Performance: O3SLM achieves state-of-the-art (SOTA) performance on multiple sketch-based tasks, including object localization, counting, image retrieval (SBIR), and VQA, outperforming existing open-weight LVLMs.
  • Significant Gains in SBIR: The paper shows substantial gains in Sketch-Based Image Retrieval (SBIR) when using O3SLM compared to baselines like LLaVA-1.5, indicating a much better understanding of sketch-image alignment. For example, on the Sketchy dataset, O3SLM-7B achieves Acc@10 of 33.2, while LLaVA-1.5-7B gets only 2.8.
  • Robustness Across Sketch Styles: O3SLM demonstrates good generalization to unseen sketch styles, as indicated by its performance on TU-Berlin sketches, which are known for their high abstraction.
  • Minimal Impact on Image-Only Tasks: O3SLM shows less than a 5% decrease in performance on standard image-only benchmarks compared to its base model (LLaVA-v1.5), demonstrating that sketch-aware training doesn't significantly degrade general vision capabilities.

Limitations and Future Work

  • Limitations: The paper acknowledges that O3SLM can struggle in scenarios with many objects that have high overlap, sometimes generating extra bounding boxes. This is a common issue in object detection tasks.
  • Future Work: The authors aim to unlock the potential of LVLMs to reason with sketches as fluently as natural images by releasing both the dataset and the model. They suggest that their approach can be extended to other abstract visual modalities.

Summary updated

Abstract

While Large Vision Language Models (LVLMs) are increasingly deployed in real-world applications, their ability to interpret abstract visual inputs remains limited. Specifically, they struggle to comprehend hand-drawn sketches, a modality that offers an intuitive means of expressing concepts that are difficult to describe textually. We identify the primary bottleneck as the absence of a large-scale dataset that jointly models sketches, photorealistic images, and corresponding natural language instructions. To address this, we present two key contributions: (1) a new, large-scale dataset of image-sketch-instruction triplets designed to facilitate both pretraining and instruction tuning, and (2) O3SLM, an LVLM trained on this dataset. Comprehensive evaluations on multiple sketch-based tasks: (a) object localization, (b) counting, (c) image retrieval i.e., (SBIR and fine-grained SBIR), and (d) visual question answering (VQA); while incorporating the three existing sketch datasets, namely QuickDraw!, Sketchy, and Tu-Berlin, along with our generated SketchVCL dataset, show that O3SLM achieves state-of-the-art performance, substantially outperforming existing LVLMs in sketch comprehension and reasoning.
BibTeX
@inproceedings{gupta2026o,
  author = {Rishi Gupta and Mukilan Karuppasamy and Shyam Marjit and Aditay Tripathi and Anirban Chakraborty},
  title = {{O3SLM: Open Weight, Open Data, and Open Vocabulary Sketch-Language Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i6.42450},
}
Figure