VisEval: A Benchmark for Data Visualization in the Era of Large Language Models

Nan Chen, Yuge Zhang, Jiahang Xu, Kan Ren, Yuqing Yang

TL;DR

VisEval is a new, rigorous benchmark designed to evaluate how well Large Language Models (LLMs) translate natural language queries into data visualizations. Unlike previous benchmarks that only check if code runs, VisEval introduces a multi-dimensional automated framework that scrutinizes the validity of the code, the legality of the chart's data mapping, and the readability of the final output.

What Problem Does This Paper Address?

Translating natural language to visualization (NL2VIS) is a complex pipeline: a model must understand the user's intent, perform data transformations (like grouping or filtering), and map that data to visual channels (axes, colors, sizes).

While LLMs like GPT-4 have shown promise in generating visualization code, existing benchmarks are often too small, contain "noisy" or incorrect labels, or focus strictly on whether the code executes. This ignores a critical reality: a chart can be "correct" in terms of code but "illegal" (mapping the wrong columns) or "unreadable" (text overlapping, or using floating-point numbers to count people). The authors identify a "gap" where current models produce charts that look okay at a glance but are fundamentally misleading or poorly designed.

Methods

The researchers developed a three-tiered automated evaluation pipeline to move beyond simple code-execution checks:

  1. Validity Checker: Ensures the generated Python code (Matplotlib or Seaborn) executes without crashing and actually calls a rendering function like plt.show().
  2. Legality Checker: This is the "logic" check. It deconstructs the generated SVG (Scalable Vector Graphics) file to extract the underlying data and chart type. It then compares these against the ground truth to ensure the model used the right columns, the right chart type (e.g., bar vs. scatter), and the correct sorting order.
  3. Readability Evaluator: Leveraging the vision capabilities of GPT-4V, this module assesses the visual quality. It checks for "layout" issues (text overflowing the canvas) and "scale/tick" issues (e.g., using a non-zero baseline for bar charts or confusing decimal ticks for integer counts).

Datasets

The authors introduced the VisEval dataset, which is a significantly cleaned and augmented version of the existing nvBench.

  • Scale: 2,524 representative queries across 146 different databases.
  • Quality Control: They used a "majority vote" from three state-of-the-art LLMs plus human expert review to filter out ambiguous, irrational, or duplicate queries found in older datasets.
  • Meta-Information: Each entry is paired with rich metadata specifying required channels, sorting requirements, and whether a "stacked bar" is strictly necessary, allowing for more nuanced evaluation than simple "exact match" tests.

Key Insights and Ideas

  • The "Three-Channel" Hurdle: LLMs perform significantly worse on charts requiring three visual channels (like stacked bar charts or grouped scatter plots) compared to simpler two-channel charts (standard bar or line charts). Complexity in mapping is a major bottleneck.
  • Seaborn vs. Matplotlib Paradox: Surprisingly, even though Seaborn is a higher-level library designed to be easier for humans, LLMs (especially GPT-4) often had higher invalid code rates when using it compared to Matplotlib. This suggests that the pre-training data for these models may be more heavily weighted toward Matplotlib.
  • Readability is the Final Boss: Even when GPT-4 produces "legal" charts, it still frequently encounters "overflow" issues where labels or legends extend beyond the image canvas, impacting real-world usability.
  • Prompting Matters: The researchers found that different LLMs have distinct "preferences" for how data tables are described in the prompt (JSON vs. Markdown vs. Natural Language), which can swing the "pass rate" by several percentage points.

Key Results

The study evaluated GPT-4, GPT-3.5, Gemini-Pro, and CodeLlama-7B:

  • GPT-4 is the leader: It achieved the highest "Quality Score" (2.89/5 in Matplotlib) and the highest pass rate (~75%). However, a 75% pass rate means it still fails to produce a "legal" chart one out of every four times.
  • CodeLlama-7B struggles: The open-source model had a high "Invalid Rate" of nearly 43%, frequently producing code that simply wouldn't run.
  • Sorting is hard: "Illegal Order" (failing to sort the data as requested) was a common error across all models, often because the models sorted by the wrong axis or failed to handle data transformations correctly before plotting.

Limitations and Future Work

The authors acknowledge several areas for growth:

  • Library Support: Currently, the benchmark focuses on Python (Matplotlib/Seaborn). Future versions could expand to JavaScript-based tools like Vega-Lite or D3.js.
  • Aesthetics and Expressiveness: The current "Readability" check focuses on fundamental errors (like overlap). It does not yet evaluate higher-level design choices like color harmony or the "storytelling" aspect of a visualization.
  • Complex Queries: The dataset focuses on common chart types. Future iterations aim to include more complex, real-world analytical queries and multi-view visualizations.

Summary updated

Abstract

Translating natural language to visualization (NL2VIS) has shown great promise for visual data analysis, but it remains a challenging task that requires multiple low-level implementations, such as natural language processing and visualization design. Recent advancements in pre-trained large language models (LLMs) are opening new avenues for generating visualizations from natural language. However, the lack of a comprehensive and reliable benchmark hinders our understanding of LLMs' capabilities in visualization generation. In this paper, we address this gap by proposing a new NL2VIS benchmark called VisEval. Firstly, we introduce a high-quality and large-scale dataset. This dataset includes 2,524 representative queries covering 146 databases, paired with accurately labeled ground truths. Secondly, we advocate for a comprehensive automated evaluation methodology covering multiple dimensions, including validity, legality, and readability. By systematically scanning for potential issues with a number of heterogeneous checkers, VisEval provides reliable and trustworthy evaluation outcomes. We run VisEval on a series of state-of-the-art LLMs. Our evaluation reveals prevalent challenges and delivers essential insights for future advancements.
BibTeX
@article{chen2025viseval,
  author = {Nan Chen and Yuge Zhang and Jiahang Xu and Kan Ren and Yuqing Yang},
  title = {{VisEval: A Benchmark for Data Visualization in the Era of Large Language Models}},
  journal = {IEEE Trans. Vis. Comput. Graph.},
  year = {2025},
  url = {https://openreview.net/forum?id=Y4exl9JOFC},
}

Loading related content...

Figure