Abstract
Kanika Goswami, Puneet Mathur, Ryan A. Rossi, Franck Dernoncourt, Vivek Gupta, Dinesh Manocha. Proceedings of The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics: System Demonstrations.
2025.
Introduction
As Large Language Models (LLMs) increasingly automate the generation of code for data visualizations, evaluating the quality of the resulting charts has become a significant challenge. Existing evaluation methods often fall into two extremes: pixel-based metrics (like SSIM) are overly sensitive to non-semantic stylistic changes, while data-centric metrics (like SCRM) ignore visual design integrity. ChartEval addresses this gap by proposing a structural evaluation framework that decomposes charts into hierarchical scene graphs, allowing for a semantic comparison that aligns more closely with human perception.
Methodology: Scene Graph Parsing
The core innovation of ChartEval is the ChartSceneParse process. Instead of comparing raw pixels, the system uses multimodal LLMs (such as GPT-4V or Claude) to extract a structured scene graph from a chart image. This graph, formatted using Vega JSON grammar, explicitly defines the chart's components, including mark types, axes, legends, and data points.
To ensure the fidelity of these graphs, ChartEval employs a self-reflection loop. The extracted JSON is rendered back into a reconstructed image using the Altair API. If the reconstruction deviates from the original data table, the system iteratively refines the graph until a high match score is achieved. Once the graphs for both the generated and ground-truth charts are finalized, the system computes four primary metrics:
- Graph-BERTScore: Semantic similarity between chart components.
- Correctness & Completeness: Precision and recall for graph nodes.
- Hallucination Rate: Identification of spurious visual elements.
- Omission Rate: Detection of missing critical information.
The ChartGen Benchmark
To validate the framework, the researchers introduced ChartGen, a comprehensive benchmark comprising 4,000 diverse chart images. This dataset integrates four existing sources—ChartCraft, ChartMimic, ChartX, and Text2Chart31—covering everything from standard 2D bar charts to complex 3D volumetric visualizations. Each entry is paired with human quality ratings to serve as a gold standard for evaluation.
Experimental Results
ChartEval significantly outperforms traditional baselines across the ChartGen benchmark. Experimental data indicates that the framework achieves a much higher Pearson correlation with human judgment than GPT-Score, SSIM, or SCRM.
As shown in the performance analysis, ChartEval remains robust even when rendering libraries or fonts change, as it focuses on the underlying data-to-visual topology rather than exact pixel values. This structural approach provides granular, interpretable feedback, allowing developers to identify specific failures such as hallucinated data points or omitted axes.
Conclusion and Future Work
ChartEval represents a shift toward semantic, structure-aware evaluation for data visualizations. By leveraging scene graphs, it provides a more accurate and interpretable metric for AI-generated charts. While currently limited by the resolution constraints of underlying vision-language models, future work aims to fine-tune models specifically for low-resolution chart parsing and expand the framework to support interactive and animated visualizations.
Summary updated