TL;DR
VDE Bench is the first systematic benchmark designed to evaluate how well image editing models manipulate dense, bilingual (Chinese-English) visual documents like academic papers, presentation slides, and exams. By shifting evaluation from standard pixel-level image metrics to fine-grained OCR parsing, the benchmark reveals that current state-of-the-art models struggle significantly with structural table edits and non-Latin scripts.
What Problem Does This Paper Address?
While multimodal image editing models have made massive strides in manipulating natural scenes or sparse text (like posters), editing dense visual documents remains a major challenge. Imagine trying to change a single sentence in a scanned academic paper or updating a value inside a complex table while perfectly preserving the original font style, background texture, and overall layout alignment.
Existing benchmarks primarily focus on English text and sparse layouts. Furthermore, standard evaluation metrics like SSIM or LPIPS operate at the image pixel level, completely missing whether the text was actually modified accurately or if the underlying document structure was broken. To bridge this gap, the authors tackle the challenge of dense, structurally complex document editing across bilingual scenarios.
Methods
The core innovation of VDE Bench is its evaluation framework, which shifts the focus from holistic image quality to fine-grained OCR parsing.
The framework evaluates models across two distinct dimensions:
- Local-image setting: Evaluates the model's precision strictly within the cropped target region intended for editing.
- Global-image setting: Evaluates the entire page to ensure the model didn't introduce unintended artifacts or corrupt unedited regions.
To quantify performance, the authors introduce a suite of complementary metrics:
- Spatial Localization: Intersection over Union (IoU) to verify if the model modified the correct bounding box area.
- Text Content Accuracy: Character Distance Metric (CDM) for character-level precision, BLEU-4 for linguistic fluency, and a tree-edit-distance (TEDS-like) metric to evaluate structural layout integrity (crucial for tables).
- Visual Fidelity: Standard image metrics (PSNR, SSIM, LPIPS, CLIP) to ensure style consistency.
Datasets
The authors introduce VDE Bench (Visual Doc Edit Bench), a dataset containing 942 high-quality, instruction-based image editing samples (comprising 674 text edits and 268 table edits).
The dataset was constructed using a rigorous four-stage pipeline:
- Data Sourcing: Seed images were curated from Omni Doc Bench, covering diverse layouts like academic papers, textbooks, exams, and newspapers.
- Instruction & Image Generation: LLMs (Gemini3-Pro and Qwen3-VL) generated 6,748 initial raw instructions across categories like text replacement, insertion, deletion, and table modification. Candidate images were generated using Nano Banana Pro.
- Human Review & Filtering: 15 computer science students manually filtered the data based on instruction compliance, realism, and subjective satisfaction, narrowing the set down to the highest quality samples.
- OCR Processing: Bounding boxes and text ground truths were structured using PaddleOCR-VL.
Key Insights and Ideas
- The Global vs. Local Paradox: A model can be excellent at modifying a localized crop but fail catastrophically when rendering the full page. Evaluating globally penalizes models that introduce unintended changes to untouched regions.
- Severe Locality Bias: Some models (like Step1x) perform exceptionally well at in-place text replacement locally but drop to near-zero accuracy on text deletion or table formatting because they fail to reason about layout reorganization.
- The Non-Latin Script Bottleneck: Models show a massive performance drop when editing Chinese text compared to English. Interestingly, this drop is concentrated on semantic accuracy and structural fidelity rather than spatial localization, meaning the models know where to edit but fail at what to write.
- Sensitivity to Ambiguity: Even a minor increase (~0.5 score) in instruction ambiguity causes an average performance drop of about 10% across all models, highlighting the need for highly explicit prompts in document editing tasks.
Key Results
The authors benchmarked several prominent models, including Longcat, Step1x, FireRed, Qwen, InstructPix2Pix, and ICEdit:
- Language Performance Gap: On the English subset, the best model achieved a BLEU score of 55.1% (FireRed). On the Chinese subset, the top score plummeted to just 16.9% (Qwen)—a relative performance drop of nearly 70%.
- Table Structure Editing: Modifying tables proved to be the hardest task. Longcat emerged as the most robust model for structural table changes, leading with a local IoU of 0.682, while Qwen excelled at preserving tree-structured layouts (achieving the highest TEDS score of 0.805).
- Human Alignment: The automated OCR evaluation metrics demonstrated a high correlation ($p = 0.94$) with human rankings, validating the reliability of the automated pipeline.
Limitations and Future Work
While VDE Bench provides a solid foundation, the authors note a few limitations:
- Language Scope: The dataset is currently restricted to English and Chinese, meaning it does not capture the unique challenges of other non-Latin scripts like Arabic, Japanese, or Korean.
- Generation Bias: The initial ground-truth images were generated using a single backbone model (Nano Banana Pro). Although strict human filtering and rejection sampling were applied to enforce a target quality distribution, some residual model bias might remain.
- Future Directions: The authors call for the community to focus on improving the structural reasoning capabilities of image editing models, particularly for complex tabular data and multi-column document layouts.
Summary updated