ChartEval: LLM-Driven Chart Generation Evaluation Using Scene Graph Parsing

Kanika Goswami, Puneet Mathur, Ryan A. Rossi, Franck Dernoncourt, Vivek Gupta, Dinesh Manocha2025 · DOI

Abstract

Kanika Goswami, Puneet Mathur, Ryan A. Rossi, Franck Dernoncourt, Vivek Gupta, Dinesh Manocha. Proceedings of The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations.

2025.

Introduction

As Large Language Models (LLMs) increasingly automate the generation of code for data visualizations, evaluating the quality of the resulting charts has become a significant challenge. Existing evaluation methods often fall into two extremes: pixel-based metrics (like SSIM) are overly sensitive to non-semantic stylistic changes, while data-centric metrics (like SCRM) ignore visual design integrity. ChartEval addresses this gap by proposing a structural evaluation framework that decomposes charts into hierarchical scene graphs, allowing for a semantic comparison that aligns more closely with human perception.

Methodology: Scene Graph Parsing

The core innovation of ChartEval is the ChartSceneParse process. Instead of comparing raw pixels, the system uses multimodal LLMs (such as GPT-4V or Claude) to extract a structured scene graph from a chart image. This graph, formatted using Vega JSON grammar, explicitly defines the chart's components, including mark types, axes, legends, and data points.

Overview of the ChartEval evaluation pipeline using scene graph decomposition to compare predicted and reference charts.

To ensure the fidelity of these graphs, ChartEval employs a self-reflection loop. The extracted JSON is rendered back into a reconstructed image using the Altair API. If the reconstruction deviates from the original data table, the system iteratively refines the graph until a high match score is achieved. Once the graphs for both the generated and ground-truth charts are finalized, the system computes four primary metrics:

  • Graph-BERTScore: Semantic similarity between chart components.
  • Correctness & Completeness: Precision and recall for graph nodes.
  • Hallucination Rate: Identification of spurious visual elements.
  • Omission Rate: Detection of missing critical information.

The ChartGen Benchmark

To validate the framework, the researchers introduced ChartGen, a comprehensive benchmark comprising 4,000 diverse chart images. This dataset integrates four existing sources—ChartCraft, ChartMimic, ChartX, and Text2Chart31—covering everything from standard 2D bar charts to complex 3D volumetric visualizations. Each entry is paired with human quality ratings to serve as a gold standard for evaluation.

Experimental Results

ChartEval significantly outperforms traditional baselines across the ChartGen benchmark. Experimental data indicates that the framework achieves a much higher Pearson correlation with human judgment than GPT-Score, SSIM, or SCRM.

Comparative performance of ChartEval metrics across multiple datasets and LLM providers, highlighting superior accuracy in correctness and completeness.

As shown in the performance analysis, ChartEval remains robust even when rendering libraries or fonts change, as it focuses on the underlying data-to-visual topology rather than exact pixel values. This structural approach provides granular, interpretable feedback, allowing developers to identify specific failures such as hallucinated data points or omitted axes.

Conclusion and Future Work

ChartEval represents a shift toward semantic, structure-aware evaluation for data visualizations. By leveraging scene graphs, it provides a more accurate and interpretable metric for AI-generated charts. While currently limited by the resolution constraints of underlying vision-language models, future work aims to fine-tune models specifically for low-resolution chart parsing and expand the framework to support interactive and animated visualizations.

Summary updated

BibTeX
@misc{goswami2025charteval,
  author = {Kanika Goswami and Puneet Mathur and Ryan A. Rossi and Franck Dernoncourt and Vivek Gupta and Dinesh Manocha},
  title = {{ChartEval: LLM-Driven Chart Generation Evaluation Using Scene Graph Parsing}},
  year = {2025},
  doi = {10.18653/v1/2025.ijcnlp-demo.10},
}
Figure