TL;DR
While Large Vision-Language Models (LVLMs) excel at processing text and natural images, they remain surprisingly blind to abstract, hand-drawn sketches. This paper introduces SketchVCL, a massive multi-task dataset of image-sketch-instruction triplets, and O3SLM, the first open-weight, open-vocabulary Sketch-Language Model. O3SLM achieves state-of-the-art performance in sketch comprehension, counting, and localization, significantly outperforming closed-source giants like GPT-4o.
What Problem Does This Paper Address?
Hand-drawn sketches are an intuitive, universal way to communicate complex visual ideas or specific object attributes that are difficult to describe with text alone. However, current state-of-the-art LVLMs struggle to understand them. As shown in Figure 1, models like LLaVA and GPT-4o fail at downstream tasks like counting or identifying objects from a crude sketch query because they treat the sketch as an abstract geometric shape rather than a semantic prompt.
The authors identify the core bottleneck: the lack of a large-scale, diverse dataset that jointly models sketches, photorealistic images, and natural language instructions. Existing sketch datasets are either class-wise only (lacking paired images) or limited to simple, single-object scenes without textual annotations.
Methods
To bridge this gap, the authors developed a unified framework consisting of an automated data curation pipeline and a two-stage training strategy:
- Automated Sketch Generation Pipeline: To scale up data generation, the authors used Segment Anything 2 (SAM2) to mask object backgrounds from real images. They then passed the foreground through a Pix2Pix model (Photo2Sketch) to generate abstract sketches, which were further enhanced using morphological gradient edge detection.
- O3SLM Architecture: Built on top of LLaVA-1.5, O3SLM uses a CLIP ViT-L/336 visual backbone to encode both sketches and natural images, a 2-layer MLP connector, and a Vicuna-v1.5 LLM base. Sketch, image, and text tokens are concatenated and fed directly into the LLM, leveraging its internal self-attention mechanism.
- Two-Stage Training:
- Stage I: Sketch Alignment: Pretraining on 600k image-sketch pairs with detailed descriptive captions (generated by DeepSeek-VL2 and refined by LLaMA-3-8B Instruct) to teach the model 3-way alignment between text, sketches, and images.
- Stage II: Instruction Tuning: Fine-tuning across four downstream tasks (Counting, Object Localization, VQA, and SBIR) using task-specific natural language prefixes (e.g.,
COUNT,BBOX) to align the model's output format.
Datasets
- SketchVCL (New): A large-scale dataset containing 600k pretraining instructions derived from Objects365 and OpenImages, and 215k instruction-tuning triplets based on COCO, Pixmo-Count, and Sketchy.
- SketchMIX (New): A diverse sketch pool combining sketches from SketchVCL, QuickDraw!, and Sketchy, all mapped to the Objects365 class taxonomy to ensure broad visual diversity.
- Evaluation Benchmarks: The model was evaluated on COCO, Pixmo-Count, Sketchy, QuickDraw!, and TU-Berlin (which was intentionally excluded from training to test zero-shot style generalization).
Key Insights and Ideas
- Sketches as Semantic Queries: By aligning sketches directly with text and images during pretraining, O3SLM learns to treat a sketch as a semantic query rather than just a geometric pattern.
- Emergent Fine-Grained Cross-Modal Understanding: Even though the model wasn't explicitly trained on combined sketch-text queries for every task, the multi-task curriculum (specifically VQA acting as an auxiliary signal) allows it to handle complex, fine-grained queries like "find the object depicted by the sketch in green or with a zebra print" (see Figure 2).
- Unfreezing the Projector Matters: The authors found that tuning the multimodal projector alongside the LLM yields massive performance gains for sketch-image alignment. In fact, a 7B model with a tuned projector can outperform a 13B model with a frozen projector.
Key Results
- Sketch-Based Counting: On the challenging COCO-Count benchmark using Sketchy queries, O3SLM-13B achieved 36.7% accuracy, vastly outperforming LLaVA-1.5-13B (2.8%) and comfortably beating GPT-4o (13.9%).
- Sketch-Based Image Retrieval (SBIR): On the Sketchy dataset, O3SLM-7B achieved an Acc@1 of 65.0%, compared to just 11.0% for the baseline LLaVA-1.5-7B.
- Zero-Shot Style Generalization: When tested on the unseen TU-Berlin sketch dataset, O3SLM maintained top performance, proving robust cross-modal transfer to completely novel drawing styles.
- Preserving Image-Only Capabilities: O3SLM retains its core vision-language capabilities, showing less than a 5% performance drop on standard image-only benchmarks (like VQAv2 and MME) while significantly improving text-only object detection.
Limitations and Future Work
The authors note that O3SLM struggles in crowded scenes containing a high density of overlapping objects, where it occasionally generates extra, redundant bounding boxes. Future work will explore integrating traditional detection algorithms like non-maximal suppression (NMS) directly into the pipeline or refining the model's spatial decoding capabilities to better handle cluttered environments.
Summary updated