VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents

Hongzhu Yi, Yujia Yang, Yuanxiang Wang, Tong Li, Zhenyu Guan, Tianyu Zong, Jiahuan Chen, Chenxi Bao, Tiankun Yang, Haopeng Jin, Yixuan Yuan, Xinming Wang, Tao Yu, Ruilin Gao, Ruiwen Tao, Haijin Liang, Jin Ma, Jinwen Luo, Yeshani, Xinyu Zuo, Jungang Xu

TL;DR

VDE Bench is the first systematic benchmark designed to evaluate how well image editing models manipulate dense, bilingual (Chinese-English) visual documents like academic papers, presentation slides, and exams. By shifting evaluation from standard pixel-level image metrics to fine-grained OCR parsing, the benchmark reveals that current state-of-the-art models struggle significantly with structural table edits and non-Latin scripts.

What Problem Does This Paper Address?

While multimodal image editing models have made massive strides in manipulating natural scenes or sparse text (like posters), editing dense visual documents remains a major challenge. Imagine trying to change a single sentence in a scanned academic paper or updating a value inside a complex table while perfectly preserving the original font style, background texture, and overall layout alignment.

Existing benchmarks primarily focus on English text and sparse layouts. Furthermore, standard evaluation metrics like SSIM or LPIPS operate at the image pixel level, completely missing whether the text was actually modified accurately or if the underlying document structure was broken. To bridge this gap, the authors tackle the challenge of dense, structurally complex document editing across bilingual scenarios.

Methods

The core innovation of VDE Bench is its evaluation framework, which shifts the focus from holistic image quality to fine-grained OCR parsing.

Figure 1. Given a document D and an editing request r, the pipeline first analyzes the layout of D, then applies r to produce the edited document D′ with edited regions {ei}, and finally runs OCR to obtain outputs {Oi}. The overall quality is measured by L = ∑k λk dk(Oi, ei, D′).

The framework evaluates models across two distinct dimensions:

  • Local-image setting: Evaluates the model's precision strictly within the cropped target region intended for editing.
  • Global-image setting: Evaluates the entire page to ensure the model didn't introduce unintended artifacts or corrupt unedited regions.

To quantify performance, the authors introduce a suite of complementary metrics:

  • Spatial Localization: Intersection over Union (IoU) to verify if the model modified the correct bounding box area.
  • Text Content Accuracy: Character Distance Metric (CDM) for character-level precision, BLEU-4 for linguistic fluency, and a tree-edit-distance (TEDS-like) metric to evaluate structural layout integrity (crucial for tables).
  • Visual Fidelity: Standard image metrics (PSNR, SSIM, LPIPS, CLIP) to ensure style consistency.

Datasets

The authors introduce VDE Bench (Visual Doc Edit Bench), a dataset containing 942 high-quality, instruction-based image editing samples (comprising 674 text edits and 268 table edits).

VDE Bench uniquely focuses on document-centric visual editing with human-verified text modifications and masks.

The dataset was constructed using a rigorous four-stage pipeline:

  1. Data Sourcing: Seed images were curated from Omni Doc Bench, covering diverse layouts like academic papers, textbooks, exams, and newspapers.
  2. Instruction & Image Generation: LLMs (Gemini3-Pro and Qwen3-VL) generated 6,748 initial raw instructions across categories like text replacement, insertion, deletion, and table modification. Candidate images were generated using Nano Banana Pro.
  3. Human Review & Filtering: 15 computer science students manually filtered the data based on instruction compliance, realism, and subjective satisfaction, narrowing the set down to the highest quality samples.
  4. OCR Processing: Bounding boxes and text ground truths were structured using PaddleOCR-VL.
Figure 2. Overview of the VDE Bench construction pipeline and a sample annotation. The pipeline consists of four stages: (1) collecting diverse documents across multiple categories and languages; (2) generating editing instructions via Gemini3-Pro / Qwen3-VL and producing edited images with Nano Banana Pro; (3) human review and annotation by 15 annotators with multi dimensional scoring, IoU based region verification, and subjective quality filtering; (4) structured OCR processing with PaddleOCR-VL and human verification. The final benchmark contains 942 high quality instruction editing samples with rich structural annotations (right).

Key Insights and Ideas

  • The Global vs. Local Paradox: A model can be excellent at modifying a localized crop but fail catastrophically when rendering the full page. Evaluating globally penalizes models that introduce unintended changes to untouched regions.
  • Severe Locality Bias: Some models (like Step1x) perform exceptionally well at in-place text replacement locally but drop to near-zero accuracy on text deletion or table formatting because they fail to reason about layout reorganization.
  • The Non-Latin Script Bottleneck: Models show a massive performance drop when editing Chinese text compared to English. Interestingly, this drop is concentrated on semantic accuracy and structural fidelity rather than spatial localization, meaning the models know where to edit but fail at what to write.
  • Sensitivity to Ambiguity: Even a minor increase (~0.5 score) in instruction ambiguity causes an average performance drop of about 10% across all models, highlighting the need for highly explicit prompts in document editing tasks.

Key Results

The authors benchmarked several prominent models, including Longcat, Step1x, FireRed, Qwen, InstructPix2Pix, and ICEdit:

  • Language Performance Gap: On the English subset, the best model achieved a BLEU score of 55.1% (FireRed). On the Chinese subset, the top score plummeted to just 16.9% (Qwen)—a relative performance drop of nearly 70%.
  • Table Structure Editing: Modifying tables proved to be the hardest task. Longcat emerged as the most robust model for structural table changes, leading with a local IoU of 0.682, while Qwen excelled at preserving tree-structured layouts (achieving the highest TEDS score of 0.805).
  • Human Alignment: The automated OCR evaluation metrics demonstrated a high correlation ($p = 0.94$) with human rankings, validating the reliability of the automated pipeline.

Limitations and Future Work

While VDE Bench provides a solid foundation, the authors note a few limitations:

  • Language Scope: The dataset is currently restricted to English and Chinese, meaning it does not capture the unique challenges of other non-Latin scripts like Arabic, Japanese, or Korean.
  • Generation Bias: The initial ground-truth images were generated using a single backbone model (Nano Banana Pro). Although strict human filtering and rejection sampling were applied to enforce a target quality distribution, some residual model bias might remain.
  • Future Directions: The authors call for the community to focus on improving the structural reasoning capabilities of image editing models, particularly for complex tabular data and multi-column document layouts.

Summary updated

Abstract

In recent years, image editing models have made significant progress, enabling users to manipulate visual content in a flexible and interactive manner through natural language instructions. However, an important yet underexplored research direction remains dense visual document image editing, which involves modifying textual content within images while faithfully preserving the original text style and background context. Existing methods primarily focus on English scenarios and images with relatively sparse text, and thus cannot adequately address dense, structurally complex documents or non-Latin scripts such as Chinese. To bridge this gap, we propose VDE Bench (Visual Doc Edit Bench), a rigorously human annotated and evaluated benchmark specifically designed to assess the performance of image editing models on bilingual Chinese-English and complex visual document editing tasks. The benchmark comprises a high quality dataset of 942 instruction based image editing samples, whose seed images encompass dense Chinese and English text documents including academic papers, posters, presentation slides, examination materials, and newspapers. Furthermore, we introduce a novel evaluation framework that systematically quantifies editing performance at the OCR parsing level, thereby enabling fine grained assessment of text modification accuracy. Based on this benchmark, we conduct a comprehensive evaluation of representative image editing models. Human verification demonstrates a high degree of consistency between human judgments and automated evaluation metrics. VDE Bench constitutes the first systematic benchmark for evaluating the performance of image editing models on bilingual dense text visual documents.
BibTeX
@misc{yi2026vde,
  author = {Hongzhu Yi and Yujia Yang and Yuanxiang Wang and Tong Li and Zhenyu Guan and Tianyu Zong and Jiahuan Chen and Chenxi Bao and Tiankun Yang and Haopeng Jin and Yixuan Yuan and Xinming Wang and Tao Yu and Ruilin Gao and Ruiwen Tao and Haijin Liang and Jin Ma and Jinwen Luo and Yeshani and Xinyu Zuo and Jungang Xu},
  title = {{VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents}},
  year = {2026},
  eprint = {2602.00122},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.00122},
}
Figure