TL;DR
VisEval is a new, rigorous benchmark designed to evaluate how well Large Language Models (LLMs) translate natural language queries into data visualizations. Unlike previous benchmarks that only check if code runs, VisEval introduces a multi-dimensional automated framework that scrutinizes the validity of the code, the legality of the chart's data mapping, and the readability of the final output.
What Problem Does This Paper Address?
Translating natural language to visualization (NL2VIS) is a complex pipeline: a model must understand the user's intent, perform data transformations (like grouping or filtering), and map that data to visual channels (axes, colors, sizes).
While LLMs like GPT-4 have shown promise in generating visualization code, existing benchmarks are often too small, contain "noisy" or incorrect labels, or focus strictly on whether the code executes. This ignores a critical reality: a chart can be "correct" in terms of code but "illegal" (mapping the wrong columns) or "unreadable" (text overlapping, or using floating-point numbers to count people). The authors identify a "gap" where current models produce charts that look okay at a glance but are fundamentally misleading or poorly designed.
Methods
The researchers developed a three-tiered automated evaluation pipeline to move beyond simple code-execution checks:
- Validity Checker: Ensures the generated Python code (Matplotlib or Seaborn) executes without crashing and actually calls a rendering function like
plt.show(). - Legality Checker: This is the "logic" check. It deconstructs the generated SVG (Scalable Vector Graphics) file to extract the underlying data and chart type. It then compares these against the ground truth to ensure the model used the right columns, the right chart type (e.g., bar vs. scatter), and the correct sorting order.
- Readability Evaluator: Leveraging the vision capabilities of GPT-4V, this module assesses the visual quality. It checks for "layout" issues (text overflowing the canvas) and "scale/tick" issues (e.g., using a non-zero baseline for bar charts or confusing decimal ticks for integer counts).
Datasets
The authors introduced the VisEval dataset, which is a significantly cleaned and augmented version of the existing nvBench.
- Scale: 2,524 representative queries across 146 different databases.
- Quality Control: They used a "majority vote" from three state-of-the-art LLMs plus human expert review to filter out ambiguous, irrational, or duplicate queries found in older datasets.
- Meta-Information: Each entry is paired with rich metadata specifying required channels, sorting requirements, and whether a "stacked bar" is strictly necessary, allowing for more nuanced evaluation than simple "exact match" tests.
Key Insights and Ideas
- The "Three-Channel" Hurdle: LLMs perform significantly worse on charts requiring three visual channels (like stacked bar charts or grouped scatter plots) compared to simpler two-channel charts (standard bar or line charts). Complexity in mapping is a major bottleneck.
- Seaborn vs. Matplotlib Paradox: Surprisingly, even though Seaborn is a higher-level library designed to be easier for humans, LLMs (especially GPT-4) often had higher invalid code rates when using it compared to Matplotlib. This suggests that the pre-training data for these models may be more heavily weighted toward Matplotlib.
- Readability is the Final Boss: Even when GPT-4 produces "legal" charts, it still frequently encounters "overflow" issues where labels or legends extend beyond the image canvas, impacting real-world usability.
- Prompting Matters: The researchers found that different LLMs have distinct "preferences" for how data tables are described in the prompt (JSON vs. Markdown vs. Natural Language), which can swing the "pass rate" by several percentage points.
Key Results
The study evaluated GPT-4, GPT-3.5, Gemini-Pro, and CodeLlama-7B:
- GPT-4 is the leader: It achieved the highest "Quality Score" (2.89/5 in Matplotlib) and the highest pass rate (~75%). However, a 75% pass rate means it still fails to produce a "legal" chart one out of every four times.
- CodeLlama-7B struggles: The open-source model had a high "Invalid Rate" of nearly 43%, frequently producing code that simply wouldn't run.
- Sorting is hard: "Illegal Order" (failing to sort the data as requested) was a common error across all models, often because the models sorted by the wrong axis or failed to handle data transformations correctly before plotting.
Limitations and Future Work
The authors acknowledge several areas for growth:
- Library Support: Currently, the benchmark focuses on Python (Matplotlib/Seaborn). Future versions could expand to JavaScript-based tools like Vega-Lite or D3.js.
- Aesthetics and Expressiveness: The current "Readability" check focuses on fundamental errors (like overlap). It does not yet evaluate higher-level design choices like color harmony or the "storytelling" aspect of a visualization.
- Complex Queries: The dataset focuses on common chart types. Future iterations aim to include more complex, real-world analytical queries and multi-view visualizations.
Summary updated