PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Retrieval Feedback

Kanika Goswami, Puneet Mathur, Ryan A. Rossi, Franck DernoncourtWWW 2025 · DOI

Introduction

Scientific data visualization is a cornerstone of research communication, yet mastering tools like Matplotlib or Seaborn remains a significant barrier for many researchers. While Large Language Models (LLMs) have shown promise in generating plotting code from natural language, they often suffer from "visual hallucinations"—producing code that executes but results in charts with overlapping text, incorrect data mapping, or poor aesthetic choices. PlotGen addresses these challenges by introducing a multi-agent framework that incorporates multimodal feedback to iteratively refine visualization code.

The PlotGen Framework

The core innovation of PlotGen is its multi-agent architecture, which mimics the human process of drafting, inspecting, and revising a plot. The system orchestrates five specialized agents to ensure high-fidelity outputs.

The PlotGen multi-agent architecture, illustrating the flow from user request to final output via specialized feedback agents.

The process begins with the Query Planning Agent, which decomposes user requests into structured plans. The Code Generation Agent then translates these plans into Python scripts. To ensure accuracy, PlotGen employs three distinct Multimodal Feedback Agents powered by GPT-4V:

  1. Numeric Feedback Agent: Verifies data consistency by comparing "de-rendered" plot points against the raw dataset.
  2. Lexical Feedback Agent: Checks for textual accuracy in titles, legends, and axis labels.
  3. Visual Feedback Agent: Evaluates aesthetic elements such as color schemes, layout, and spatial alignment.

These agents provide targeted critiques, allowing the Code Generation Agent to self-correct until the visualization meets the required standards.

Evaluation and Results

PlotGen was evaluated using MatPlotBench, a benchmark comprising 100 complex scientific plotting scenarios. The framework demonstrated superior performance across various LLM backbones, including both proprietary and open-source models.

Quantitative comparison of PlotGen against baselines across different LLM backbones on the MatPlotBench dataset.

As shown in the results table, PlotGen consistently outperformed state-of-the-art methods like MatPlotAgent. When using GPT-4, PlotGen achieved a score of 65.67, representing a significant improvement over direct decoding and zero-shot Chain-of-Thought (CoT) approaches. Ablation studies further highlighted the importance of visual feedback; removing the Visual Feedback Agent led to a 10-15% drop in performance, confirming that "seeing" the plot is essential for resolving layout and aesthetic issues.

In a qualitative user study, participants rated 40.5% of PlotGen's outputs as "Completely Accurate" and 24.5% as "Somewhat Accurate." Users particularly valued the system's ability to automate the tedious debugging process, significantly reducing the manual effort required to produce publication-quality figures.

Conclusion

PlotGen represents a significant step forward in automated scientific visualization. By integrating multimodal self-reflection into the code generation pipeline, it bridges the gap between raw data and accurate, aesthetically pleasing charts. Future work will focus on expanding the framework's capabilities to interactive dashboards and more complex 3D environments.

Summary updated

BibTeX
@inproceedings{goswami2025plotgen,
  author = {Kanika Goswami and Puneet Mathur and Ryan A. Rossi and Franck Dernoncourt},
  title = {{PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Retrieval Feedback}},
  booktitle = {The Web Conference},
  year = {2025},
  doi = {10.1145/3701716.3716888},
}
Figure