Abstract
Translating natural language to visualization (NL2VIS) has shown great promise for visual data analysis, but it remains a challenging task that requires multiple low-level implementations, such as natural language processing and visualization design. Recent advancements in pre-trained large language models (LLMs) are opening new avenues for generating visualizations from natural language. However, the lack of a comprehensive and reliable benchmark hinders our understanding of LLMs' capabilities in visualization generation. In this paper, we address this gap by proposing a new NL2VIS benchmark called VisEval. Firstly, we introduce a high-quality and large-scale dataset. This dataset includes 2,524 representative queries covering 146 databases, paired with accurately labeled ground truths. Secondly, we advocate for a comprehensive automated evaluation methodology covering multiple dimensions, including validity, legality, and readability. By systematically scanning for potential issues with a number of heterogeneous checkers, VisEval provides reliable and trustworthy evaluation outcomes. We run VisEval on a series of state-of-the-art LLMs. Our evaluation reveals prevalent challenges and delivers essential insights for future advancements.
Introduction
The task of translating natural language to visualization (NL2VIS) is a complex pipeline. A model must not only understand the user's intent but also perform necessary data transformations—such as grouping, filtering, or sorting—and accurately map that data to visual channels like axes, colors, and sizes. While Large Language Models (LLMs) like GPT-4 have demonstrated remarkable code-generation capabilities, existing benchmarks for NL2VIS often fall short. Many are too small, contain noisy labels, or focus strictly on whether the generated code executes. This narrow focus ignores a critical reality: a chart can be syntactically correct in terms of code but fundamentally "illegal" (mapping the wrong data) or "unreadable" (containing overlapping text or misleading scales).
VisEval addresses these gaps by introducing a rigorous, multi-dimensional automated framework that scrutinizes the validity, legality, and readability of LLM-generated visualizations.
The VisEval Evaluation Framework
To move beyond simple code-execution checks, the researchers developed a three-tiered automated evaluation pipeline:
- Validity Checker: This stage ensures the generated Python code (typically using Matplotlib or Seaborn) executes without crashing and includes necessary rendering functions like
plt.show(). - Legality Checker: This module performs a logic check by deconstructing the generated SVG (Scalable Vector Graphics) file. It extracts the underlying data and chart type to compare them against the ground truth. This ensures the model used the correct columns, chart type (e.g., bar vs. scatter), and sorting order.
- Readability Evaluator: Leveraging the vision capabilities of GPT-4V, this module assesses visual quality. It identifies layout issues such as text overflowing the canvas and scale/tick errors, such as using non-zero baselines for bar charts or confusing decimal ticks for integer counts.
The VisEval Dataset
The authors introduced the VisEval dataset, a significantly cleaned and augmented version of the existing nvBench. The dataset features 2,524 representative queries across 146 different databases. To ensure high quality, the researchers employed a "majority vote" from three state-of-the-art LLMs followed by human expert review to filter out ambiguous or irrational queries. Each entry is paired with rich metadata specifying required channels and sorting requirements, allowing for a more nuanced evaluation than simple "exact match" tests.
Key Insights and Experimental Results
The study evaluated several prominent models, including GPT-4, GPT-3.5, Gemini-Pro, and CodeLlama-7B. Several critical insights emerged from the benchmarking process:
- The "Three-Channel" Hurdle: LLMs perform significantly worse on charts requiring three visual channels (such as stacked bar charts or grouped scatter plots) compared to simpler two-channel charts. Mapping complexity remains a major bottleneck.
- Matplotlib vs. Seaborn: Surprisingly, LLMs often had higher invalid code rates when using Seaborn compared to Matplotlib, despite Seaborn being a higher-level library. This suggests that LLM training data may be more heavily weighted toward Matplotlib.
- Readability Challenges: Even when models produce "legal" charts, they frequently encounter "overflow" issues where labels or legends extend beyond the image canvas. Additionally, models often struggle with "Illegal Order," failing to sort data as requested because they sort by the wrong axis or fail to handle data transformations correctly.
GPT-4 emerged as the clear leader with a Quality Score of 2.89/5 and a pass rate of approximately 75%. However, this still implies a failure rate of one in four, highlighting the need for further advancement in NL2VIS capabilities.
Conclusion and Future Work
VisEval provides a much-needed benchmark for the rigorous evaluation of LLMs in data visualization. While current models show promise, significant hurdles remain in handling complex mappings and ensuring visual readability. Future work will aim to expand the benchmark to include JavaScript-based libraries like Vega-Lite or D3.js and to evaluate higher-level design choices such as color harmony and the narrative "storytelling" aspect of visualizations.
Summary updated