TL;DR
This paper addresses the limitations of current Vision-Language Models (VLLMs) in understanding spatial relationships by introducing a new, expanded benchmark and a novel VLLM architecture called VSRE. VSRE significantly improves VLLMs' ability to accurately interpret spatial rules, overcoming issues like over-sensitivity to text and under-sensitivity to visual cues.
What Problem Does This Paper Address?
Reasoning about spatial relationships is fundamental to human cognition, yet current Vision-Language Models (VLLMs) struggle with this. While existing benchmarks like MME, MMBench, and SEED have touched upon visual spatial reasoning (VSR), there's a notable lack of comprehensive evaluation and optimization datasets specifically for VLLMs in this domain. The authors observed that existing VLLMs exhibit a peculiar behavior: they are overly sensitive to the nuances of language instructions but insufficiently attentive to the visual positional information presented. This leads to inconsistent performance, answer biases, and an inability to truly "see" spatial relationships. The paper aims to bridge this gap by creating a more robust VSR benchmark and developing a VLLM that excels in this area.
Methods
The authors tackle the VSR problem through a multi-pronged approach:
- Unified Instruction Test Set: They created a new VSR test set by expanding the original VSR dataset with both manual and GPT-4 generated templates. This aims to standardize evaluation and address inconsistencies.
- Data Expansion:
- Text Data: The training data was significantly expanded by rewriting existing text captions and generating new question-answer pairs using manual and GPT-4 templates. This aims to improve generalization to various question formats.
- Image Data: Diffusion models were employed to controllably augment and repaint visual training images with specific spatial relation concepts (e.g., "in," "on," "under"). This includes techniques like image-to-image translation, text-to-image generation, and image inpainting to create diverse and spatially rich visual data.
- Merged Vision Encoder: To enhance visual perception, the authors integrated multiple powerful visual encoders (CLIP, SigLIP, DINOv2, and SAM) into a merged vision encoder. This aims to capture finer visual details crucial for spatial reasoning.
- VSRE Architecture: The proposed VLLM, named VSRE (Visual Spatial Reasoning Expert), is built upon the LLaVA architecture and incorporates the expanded datasets and the merged vision encoder. The training involves a two-stage process: pertaining to enhance spatial understanding and instruction fine-tuning (IFT) for better instruction following.
Datasets
- Original VSR Dataset: Used as a seed for expansion.
- New Unified VSR Instruction Test Set: Created by expanding the original VSR dataset with manual and GPT-4 generated templates for evaluation.
- Expanded Training Data: Nearly 100 times the original amount, generated through text rewriting and image repainting using diffusion models. This includes "pre-training data" (pre-100k, pre-500k) and "IFT data" (turn-g 500k).
- Other Benchmarks: MME, MMBench, and SEEDv2 were used for evaluating the generalization capabilities of the proposed VSRE model.
Key Insights and Ideas
- VLLMs' Spatial Blind Spots: The paper highlights a critical flaw in current VLLMs: an over-reliance on linguistic cues and a deficiency in truly understanding visual spatial relationships. This is a significant hurdle for tasks requiring fine-grained perception.
- The Power of Controlled Image Generation: Using diffusion models for data augmentation in VSR is a novel approach. It allows for precise control over spatial relationships in generated images, leading to more effective training data.
- Ensemble of Vision Encoders: Combining multiple vision encoders, especially those trained with language supervision (like SigLIP) and self-supervised methods (like DINOv2), significantly boosts performance on spatial reasoning tasks. This suggests that a diverse set of visual features is key.
- Unified Evaluation is Crucial: The lack of a standardized instruction test set for VSR has led to inconsistent results and hindered progress. The proposed unified test set is essential for reliable evaluation and comparison.
Key Results
- The proposed VSRE model achieved a significant 27% increase in accuracy on the VSR test set compared to the baseline LLaVA1.5 13B (from ~52% to ~79%).
- VSRE demonstrated superior performance on the VSR dataset and relevant subsets of other benchmarks like MME, MMBench, and SEEDv2, outperforming optimized baselines.
- The model showed improved differentiation of visual positional concepts and a better ability to summarize them, indicating a deeper understanding of spatial relationships.
- The bias issue (e.g., higher accuracy for "yes" vs. "no" questions) was significantly alleviated, with VSRE being less misled by co-occurring entities.
Limitations and Future Work
- Limitations: While the paper demonstrates significant improvements, the authors acknowledge that the accuracy gains from adding more visual features diminish as the number of features increases. The performance on "no" questions, while improved, is still lower than "yes" questions, indicating room for further bias reduction.
- Future Work: The authors suggest that their expansion method highlights the immense potential in VSR field for related data and model structures. Further research could explore more advanced vision encoder combinations and more sophisticated data augmentation strategies to push the boundaries of VLLM spatial reasoning.
Summary updated