Abstract
Unified multimodal models target joint understanding, reasoning, and generation, but current image editing benchmarks are largely confined to natural images and shallow commonsense reasoning, offering limited assessment of this capability under structured, domain-specific constraints. In this work, we introduce GRADE, the first benchmark to assess discipline-informed knowledge and reasoning in image editing. GRADE comprises 520 carefully curated samples across 10 academic domains, spanning from natural science to social science. To support rigorous evaluation, we propose a multi-dimensional evaluation protocol that jointly assesses Discipline Reasoning, Visual Consistency, and Logical Readability. Extensive experiments on 20 state-of-the-art open-source and closed-source models reveal substantial limitations in current models under implicit, knowledge-intensive editing settings, leading to large performance gaps. Beyond quantitative scores, we conduct rigorous analyses and ablations to expose model shortcomings and identify the constraints within disciplinary editing. Together, GRADE pinpoints key directions for the future development of unified multimodal models, advancing the research on discipline-informed image editing and reasoning.
Introduction
While unified multimodal models (UMMs) have demonstrated remarkable proficiency in generating and editing natural images, their performance often falters when faced with tasks requiring specialized academic knowledge. Existing benchmarks typically focus on "shallow" reasoning—for instance, following a prompt to "make a cat wear a hat." However, true multimodal intelligence requires discipline-informed reasoning: the ability to interpret structured, domain-specific constraints and modify an image while maintaining logical and visual integrity.
Consider a task requiring a model to "resolve a suspended fourth chord to a major triad" in a musical score, or "draw the shortest path in a Dijkstra’s algorithm diagram." These tasks demand more than just aesthetic generation; they require a deep coordination between specialized knowledge and precise visual synthesis. Before the introduction of GRADE, there was no rigorous framework to measure this specific intersection of reasoning and editing.
The GRADE Benchmark
The Grounded Reasoning Assessment for Discipline-informed Editing (GRADE) benchmark is designed to fill this gap. It consists of 520 carefully curated image-triplets, each containing an input image, a natural language instruction, and a ground-truth (GT) result.
The benchmark spans 10 diverse academic disciplines:
- Natural Sciences: Physics, Chemistry, Biology.
- Formal Sciences: Mathematics, Computer Science.
- Social Sciences & Humanities: History, Geography, Economics.
- Applied Arts: Music, Sports.
Data was sourced from high-quality educational resources, including textbooks and academic slides, as well as existing datasets like MMMU and Geometry3k. To ensure structural precision, several samples were programmatically generated using tools such as GeoGebra and Matplotlib.
Multi-Dimensional Evaluation
To move beyond subjective aesthetic assessments, GRADE employs a multi-dimensional evaluation pipeline that assesses three core pillars of discipline-informed editing:
- Discipline Reasoning: This metric evaluates whether the model correctly applied the required academic logic. The process uses a "judge" model (Gemini-3-Flash) guided by fine-grained, weighted scoring points generated by GPT-5. This breaks complex tasks into binary checkpoints (e.g., "Is the A-B edge colored red?").
- Visual Consistency: This assesses whether the model preserved the original structure and style in areas of the image that were not intended to be modified.
- Logical Readability: This evaluates the clarity of the final representation, ensuring that labels are correct, nodes are distinguishable, and the output follows standard academic conventions.
Experimental Results and Analysis
Extensive testing on 20 state-of-the-art models revealed a significant performance gap. Nano Banana Pro emerged as the state-of-the-art (SoTA) with a strict accuracy of 46.2%, followed by Nano Banana 2 at 39.6%.
The Reasoning Gap
A key finding of the study is the "Implicit Bottleneck." Models struggle significantly more with implicit instructions that require them to deduce the underlying logic. When these same tasks were presented with "explicit" step-by-step instructions, performance improved dramatically. For example, Nano Banana 2's reasoning score rose from 67.9% to 89.7% under explicit prompting, suggesting that the primary failure mode is often in the reasoning phase rather than the generation phase.
Open vs. Closed Source Disparity
There is a staggering disparity between proprietary and open-source models. While the top closed-source model reached 46.2%, the best-performing open-source model (Qwen-Edit-2511) achieved only 2.7%. This highlights a critical need for the open-source community to focus on reasoning-heavy multimodal training data.
Failure Mode Categorization
The authors identified four primary types of model failure:
- Perception: Misidentifying key objects (e.g., misreading a musical note or a graph node).
- Knowledge: Lacking the factual basis to solve the problem (e.g., using incorrect labels for chemical sublevels).
- Reasoning: Following a flawed logical path despite having the correct methodology.
- Generation: Failing to accurately draw the intended result even when the reasoning is correct.
Conclusion
GRADE sets a high bar for the next generation of unified multimodal models. Currently, even the most advanced models fail more than half the time when faced with strict discipline-informed constraints. Future research must focus on better knowledge integration—allowing models to "think through" structured data before generating pixels—and improving fine-grained control to ensure models do not drift from hard logical constraints during the synthesis process.
Summary updated