TL;DR
This paper introduces O3SLM, an open-weight, open-data model that significantly improves Large Vision-Language Models' (LVLMs) ability to understand hand-drawn sketches. They achieve this by releasing SketchVCL, a novel, large-scale dataset of image-sketch-instruction triplets, and demonstrating O3SLM's state-of-the-art performance on various sketch-based reasoning tasks.
What Problem Does This Paper Address?
Current Large Vision-Language Models (LVLMs) excel at understanding photorealistic images but struggle with more abstract visual inputs like hand-drawn sketches. This limitation hinders their application in scenarios where sketches are used for intuitive communication or as a more universal language. The core problem is the lack of a comprehensive, large-scale dataset that jointly models sketches, real images, and natural language instructions, which is crucial for training LVLMs to effectively interpret and reason about sketches.
Methods
The authors propose a two-stage training approach for their O3SLM model:
- Stage I: Sketch Alignment (Pretraining): This stage focuses on aligning sketches with images and text. They use a large-scale dataset (SketchVCL-OI, SketchVCL-O365) to teach the model to recognize sketches, associate them with objects in natural images, develop spatial understanding, and retain natural language capabilities. This ensures a "3-way alignment" of sketches, images, and text.
- Stage II: Instruction Tuning: This stage fine-tunes the model on specific sketch-based tasks. They train O3SLM on four tasks: Counting, Object Localization, Visual Question Answering (VQA), and Sketch-Based Image Retrieval (SBIR), using a curated task-specific dataset. They employ a prefix-based approach for task descriptors rather than modifying the vocabulary.
The O3SLM architecture itself uses a CLIP ViT-L/336 as the visual backbone, a multimodal connector (two-layered MLP) to project features into the LLM's input space, and the Vicuna v1.5 LLM as the base language model. They leverage LLaVA-1.5 weights for initialization.
Datasets
The paper introduces SketchVCL, a novel, large-scale dataset designed for sketch-image-text alignment and instruction tuning. SketchVCL is composed of image-sketch-instruction triplets and is built using an automated sketch generation pipeline. It includes:
- SketchVCL-OI and SketchVCL-O365: For pretraining, derived from OpenImages and Object365 datasets, respectively.
- SketchMIX: For instruction tuning, a mixture of sketches from various sources including SketchVCL-O365, SketchVCL-C, QuickDraw!, Sketchy, and TU-Berlin.
Existing datasets like QuickDraw!, Sketchy, and TU-Berlin are also incorporated into SketchMIX and used for evaluation.
Key Insights and Ideas
- Sketches as a Crucial Modality: The paper strongly advocates for the importance of sketches as a visual modality for LVLMs, highlighting their expressive power and ability to transcend linguistic barriers.
- The "Sketch Gap" in LVLMs: Existing LVLMs exhibit a significant performance drop when presented with sketches, indicating a clear gap in their understanding of abstract visual inputs.
- Dataset is Key: The primary bottleneck identified is the lack of a suitable large-scale dataset. The creation of SketchVCL is presented as a direct solution to this problem.
- Pretraining is Essential for Sketches: The results show that large-scale pretraining (Stage I) is crucial for achieving good performance on sketch-based tasks, particularly for SBIR, significantly boosting performance over models trained only with fine-tuning.
- Emergent Fine-Grained Understanding: O3SLM demonstrates an emergent ability to perform fine-grained SBIR tasks, even when not explicitly trained for it, by leveraging the multitask pretraining and VQA supervision.
Key Results
- State-of-the-Art Performance: O3SLM achieves state-of-the-art (SOTA) performance on multiple sketch-based tasks, including object localization, counting, image retrieval (SBIR), and VQA, outperforming existing open-weight LVLMs.
- Significant Gains in SBIR: The paper shows substantial gains in Sketch-Based Image Retrieval (SBIR) when using O3SLM compared to baselines like LLaVA-1.5, indicating a much better understanding of sketch-image alignment. For example, on the Sketchy dataset, O3SLM-7B achieves Acc@10 of 33.2, while LLaVA-1.5-7B gets only 2.8.
- Robustness Across Sketch Styles: O3SLM demonstrates good generalization to unseen sketch styles, as indicated by its performance on TU-Berlin sketches, which are known for their high abstraction.
- Minimal Impact on Image-Only Tasks: O3SLM shows less than a 5% decrease in performance on standard image-only benchmarks compared to its base model (LLaVA-v1.5), demonstrating that sketch-aware training doesn't significantly degrade general vision capabilities.
Limitations and Future Work
- Limitations: The paper acknowledges that O3SLM can struggle in scenarios with many objects that have high overlap, sometimes generating extra bounding boxes. This is a common issue in object detection tasks.
- Future Work: The authors aim to unlock the potential of LVLMs to reason with sketches as fluently as natural images by releasing both the dataset and the model. They suggest that their approach can be extended to other abstract visual modalities.
Summary updated