How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing

Huanyu Zhang, Xuehai Bai, Chengzu Li, Chen Liang, Haochen Tian, Haodong Li, Ruichuan An, Yifan Zhang, Anna Korhonen, Liang Wang, Tieniu TanarXiv 2026 · arXiv · OpenReview

TL;DR

This paper introduces VIBE, a new benchmark for evaluating image editing models that follow visual instructions (like sketches or arrows) instead of just text. They found that while proprietary models are getting better at this, they still struggle with complex instructions, highlighting a need for more advanced reasoning capabilities.

What Problem Does This Paper Address?

Current image editing models primarily rely on text-based instructions. While effective for simple edits, text alone can be ambiguous and requires users to be very descriptive, leading to a high cognitive load. Humans, however, naturally combine language with visual cues like sketches and annotations to convey precise spatial and structural intent. This multimodal approach is more intuitive and efficient. The authors identified a gap in existing benchmarks: they don't systematically evaluate how well models can follow these richer, visually-guided instructions, which are crucial for more complex and nuanced image editing tasks.

Methods

The core of this paper is the VIBE benchmark, designed to systematically evaluate visual instruction-following in image editing. VIBE is structured around a three-level interaction hierarchy:

  1. Deictic Grounding: Models identify and isolate specific locations or objects using visual cues (e.g., bounding boxes, arrows).
  2. Morphological Manipulation: Models use visual cues (e.g., sketches, skeletons) to define structural constraints for edits like changing pose or shape.
  3. Causal Reasoning: Visual instructions act as catalysts, encoding physical or logical dynamics (e.g., force vectors, motion arrows) that require models to predict outcomes.

For evaluation, they developed a Large Multimodal Model (LMM)-as-a-judge framework. This uses an LMM (GPT5.1) to assess model outputs based on task-specific metrics like Instruction Adherence, Contextual Preservation, and Visual Coherence, ensuring scalable and fine-grained assessment.

Datasets

The paper introduces the VIBE dataset, comprising 1,034 manually annotated samples across 10 diverse subtasks. These tasks are organized into the three hierarchical levels mentioned above. The dataset includes images spanning real-world, animation, and sketch styles to ensure broad coverage and test model robustness across different visual domains.

Key Insights and Ideas

  • Visual Instructions are Key: The paper strongly advocates for visual instructions as a more natural and effective way to guide image editing, reducing ambiguity and cognitive load compared to text-only methods.
  • Hierarchical Complexity Matters: The three-level hierarchy (Deictic, Morphological, Causal) effectively captures the increasing difficulty of visual instruction following, revealing distinct performance patterns across these levels.
  • LMMs as Evaluators: The use of LMMs as judges proves to be a reliable and scalable method for evaluating complex generative tasks, showing strong correlation with human expert judgments.
  • Proprietary vs. Open-Source Divide: A significant performance gap exists, with proprietary models generally outperforming open-source ones, suggesting advantages in model scale and training data.

Key Results

  • Early-Stage Competence: Proprietary models show promising capabilities in basic visual instruction following (Deictic Level), often achieving scores above 60% on tasks like addition, removal, and replacement.
  • Performance Degradation: Performance drops significantly as task complexity increases, especially at the Causal Level, where even top models struggle to achieve 50% on average. This validates the hierarchical design of VIBE.
  • Style-Specific Preferences: Models exhibit varied performance across different image styles (real-world, animation, sketch), with some models showing strong preferences or degradations on specific styles.
  • Multi-task Challenges: Models struggle with composing multiple visual instructions within a single query, showing a clear drop in performance compared to single-task execution.

Limitations and Future Work

  • Causal Reasoning Gap: The most significant challenge lies in causal reasoning, where models fail to accurately predict the outcomes of physical interactions. This indicates a need for models with better world models and reasoning capabilities.
  • Compositional Understanding: Models need to improve their ability to understand and execute multiple instructions simultaneously, a crucial aspect of real-world human-computer interaction.
  • Open-Source Advancement: The paper highlights a clear opportunity for future research to narrow the performance gap between proprietary and open-source models by focusing on instruction understanding, multimodal alignment, and advanced reasoning.
  • Further Benchmark Expansion: The authors suggest VIBE can serve as a testbed for developing models with improved visual instruction following and coordination with textual guidance.

Summary updated

Abstract

Recent generative models have achieved remarkable progress in image editing. However, existing systems and benchmarks remain largely text-guided. In contrast, human communication is inherently multimodal, where visual instructions such as sketches efficiently convey spatial and structural intent. To address this gap, we introduce VIBE, the Visual Instruction Benchmark for Image Editing with a three-level interaction hierarchy that captures deictic grounding, morphological manipulation, and causal reasoning. Across these levels, we curate high-quality and diverse test cases that reflect progressively increasing complexity in visual instruction following. We further propose a robust LMM-as-a-judge evaluation framework with task-specific metrics to enable scalable and fine-grained assessment. Through a comprehensive evaluation of 17 representative open-source and proprietary image editing models, we find that proprietary models exhibit early-stage visual instruction-following capabilities and consistently outperform open-source models. However, performance degrades markedly with increasing task difficulty even for the strongest systems, highlighting promising directions for future research.
BibTeX
@misc{zhang2026how,
  author = {Huanyu Zhang and Xuehai Bai and Chengzu Li and Chen Liang and Haochen Tian and Haodong Li and Ruichuan An and Yifan Zhang and Anna Korhonen and Zhang Zhang and Liang Wang and Tieniu Tan},
  title = {{How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing}},
  year = {2026},
  eprint = {2602.01851},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.01851},
}
Figure