Charts Are Not Images: On the Challenges of Scientific Chart Editing

Shawn Li, Ryan A. Rossi, Sungchul Kim, Sunav Choudhary, Franck Dernoncourt, Puneet Mathur, Zhengzhong Tu, Yue Zhao

TL;DR

This paper argues that editing scientific charts requires understanding their underlying data structure, not just their visual appearance. They introduce FigEdit, a large-scale benchmark with over 30,000 chart editing examples, to highlight the limitations of current image editing models and promote the development of structure-aware approaches.

What Problem Does This Paper Address?

Current AI models excel at editing natural images, often using pixel-level manipulation. However, applying these models to scientific charts is problematic because charts are more than just pixels; they are visual representations of structured data governed by specific rules (a "graphical grammar"). Instructions like "add a bar for category X with value 42" require modifying the underlying data and its visual mapping, not just rearranging pixels. Existing models often fail at this, producing outputs that look plausible but are semantically incorrect or violate visualization rules. This mismatch between current methods and the nature of chart editing is the core problem.

Methods

The authors introduce FigEdit, a novel benchmark designed specifically for scientific chart editing. FigEdit comprises over 30,000 real-world chart editing instances, covering 10 different chart types and a diverse set of complex editing instructions. The benchmark is structured into five progressively challenging tasks:

  • Single Edits: Modifying a single element or attribute.
  • Multi Edits: Applying multiple edits simultaneously.
  • Conversational Edits: Handling multi-turn editing dialogues.
  • Visual-Guidance-Based Edits: Using visual cues (like a circled region) to guide edits.
  • Style Transfer Edits: Matching the style of one chart to another.

FigEdit provides not only edited chart images but also executable specifications, enabling a more robust evaluation of semantic correctness. They also propose semantics-aware evaluation metrics that go beyond traditional pixel-similarity measures like SSIM and PSNR.

Datasets

The base figures for the FigEdit benchmark are derived from a wide range of real-world datasets spanning domains like economics, climate, healthcare, sports, and social science. The paper lists these datasets in Appendix G and H, noting their sources from public repositories, statistical agencies, and journalism archives.

Key Insights and Ideas

  • Charts are not just images: The fundamental insight is that scientific charts are structured data visualizations, and editing them requires understanding this structure, not just pixel manipulation.
  • Pixel-level metrics are insufficient: Traditional image quality metrics (SSIM, PSNR) are poor indicators of semantic correctness in chart editing. Models can achieve high scores on these metrics while failing to execute edits accurately.
  • Current models struggle with structured edits: State-of-the-art image editing models perform poorly on scientific charts, consistently failing to handle the underlying data transformations required for valid edits.
  • A new benchmark is needed: The authors argue that existing benchmarks are inadequate for evaluating chart editing, lacking diversity in chart types, instruction complexity, and semantic evaluation.

Key Results

  • Poor performance of existing models: Evaluations on FigEdit reveal that current leading models (Imagen 4, GPT-Image, OmniGen2, InstructPix2Pix) struggle significantly with scientific chart editing tasks, especially those requiring semantic consistency.
  • Limitations of pixel metrics: The paper demonstrates with examples that high SSIM/PSNR scores do not correlate with correct edits. For instance, a model might achieve a high SSIM score but fail to apply the requested edit or preserve data integrity.
  • No single model dominates: Performance is highly fragmented across models and tasks, indicating that current models often overfit to specific task structures or metric types.
  • FigEdit's utility: The benchmark effectively highlights the limitations of current approaches and provides a foundation for developing and evaluating future structure-aware models.

Limitations and Future Work

The authors acknowledge that their benchmark, while large-scale, is still a step towards addressing the complex problem of scientific chart editing. They suggest that future work should focus on developing models that can truly understand and manipulate the structured nature of charts, moving beyond pixel-level operations. This includes research into models that can reason about data, visualization grammar, and semantic consistency to produce faithful and auditable edits. The release of FigEdit aims to facilitate this progress by providing a standardized evaluation platform.

Summary updated

Abstract

Generative models, such as diffusion and autoregressive approaches, have demonstrated impressive capabilities in editing natural images. However, applying these tools to scientific charts rests on a flawed assumption: a chart is not merely an arrangement of pixels but a visual representation of structured data governed by a graphical grammar. Consequently, chart editing is not a pixel-manipulation task but a structured transformation problem. To address this fundamental mismatch, we introduce \textit{FigEdit}, a large-scale benchmark for scientific figure editing comprising over 30,000 samples. Grounded in real-world data, our benchmark is distinguished by its diversity, covering 10 distinct chart types and a rich vocabulary of complex editing instructions. The benchmark is organized into five distinct and progressively challenging tasks: single edits, multi edits, conversational edits, visual-guidance-based edits, and style transfer. Our evaluation of a range of state-of-the-art models on this benchmark reveals their poor performance on scientific figures, as they consistently fail to handle the underlying structured transformations required for valid edits. Furthermore, our analysis indicates that traditional evaluation metrics (e.g., SSIM, PSNR) have limitations in capturing the semantic correctness of chart edits. Our benchmark demonstrates the profound limitations of pixel-level manipulation and provides a robust foundation for developing and evaluating future structure-aware models. By releasing \textit{FigEdit} (https://github.com/adobe-research/figure-editing), we aim to enable systematic progress in structure-aware figure editing, provide a common ground for fair comparison, and encourage future research on models that understand both the visual and semantic layers of scientific charts.
BibTeX
@inproceedings{li2026charts,
  author = {Shawn Li and Ryan A. Rossi and Sungchul Kim and Sunav Choudhary and Franck Dernoncourt and Puneet Mathur and Zhengzhong Tu and Yue Zhao},
  title = {{Charts Are Not Images: On the Challenges of Scientific Chart Editing}},
  booktitle = {International Conference on Learning Representations},
  year = {2026},
  url = {https://openreview.net/forum?id=259xBeNyDV},
}
Figure