TL;DR
This paper introduces a lightweight multi-agent system that significantly reduces errors in text-to-chart generation, outperforming existing methods by nearly 5 percentage points. The research suggests that while execution errors are largely solved, future efforts should focus on improving chart aesthetics, semantic accuracy, and accessibility, particularly for colorblind users.
What Problem Does This Paper Address?
Generating charts from natural language descriptions is a crucial task for making data accessible to non-programmers. While large language models (LLMs) have shown promise, a significant percentage of generated code still fails to execute, breaking downstream applications. Existing benchmarks primarily focus on execution success and code similarity, potentially overlooking other critical aspects of chart quality like visual fidelity and accessibility. This paper investigates whether the persistent error rate is due to model limitations or single-prompt designs and explores what truly defines a "solved" text-to-chart generation task.
Methods
The authors propose a multi-agent pipeline that breaks down the text-to-chart generation process into distinct roles:
- DRAFTING Agent: This agent takes a natural language description and generates Python/Matplotlib code.
- RE-WRITER Agent: If the drafted code fails to execute, this agent receives the error traceback and attempts to debug and rewrite the code.
This loop repeats for up to three iterations. The system uses an off-the-shelf GPT-40-mini model, requiring no additional fine-tuning or reinforcement learning. This lightweight, agentic approach aims to improve robustness and efficiency.
Datasets
The study evaluates its approach on three established text-to-chart generation benchmarks:
- TEXT2CHART31: A large dataset with 31 plot types, including synthetic data generated by GPT-3.5-turbo and GPT-4.
- ChartX: A dataset with 18 chart types, also synthetically generated.
- VisText: A dataset of 12,441 bar, line, and area charts constructed from real-world data.
The paper notes that for ChartX and VisText, the test sets were used for evaluation, and for VisText, chart descriptions were automatically generated.
Key Insights and Ideas
- Agentic approach is key: A simple multi-agent system, even with a single off-the-shelf model, can dramatically reduce execution errors compared to single-prompt LLMs. This suggests that structured reasoning and iterative refinement are more impactful than simply using larger or fine-tuned models for this task.
- Execution is largely "solved": The significant reduction in execution errors (down to 4.5% on TEXT2CHART31) implies that current benchmarks may be too focused on this aspect, and the field needs to shift its attention.
- Beyond execution: Aesthetics and Accessibility Matter: While execution is improved, a manual review revealed that a substantial portion of generated charts suffer from stylistic deviations or, more critically, fail to meet basic colorblindness guidelines (only 7-33% were appropriate). This highlights a significant gap in current evaluation metrics and real-world usability.
- Code similarity is less important than execution: The study found that while execution success improved, code similarity metrics did not show similar gains. This suggests that for practical chart generation, ensuring the code runs and produces a visually acceptable output is more critical than strict adherence to the original code's structure.
Key Results
- The multi-agent system reduced execution errors to 4.5% on TEXT2CHART31 and 4.6% on ChartX, outperforming fine-tuned baselines by nearly 5 percentage points.
- Image quality, as measured by SSIM and multimodal LLM judgments, remained comparable between the agentic system and baselines, indicating no degradation in visual fidelity.
- Manual analysis of 100 sampled charts found 83% to be fully successful, with most remaining issues being stylistic rather than data errors.
- A significant finding was that only 33.3% of generated charts on TEXT2CHART31 and 7.2% on ChartX met basic colorblindness guidelines.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset limitations: The datasets used are either synthetically generated or have automatically created descriptions, which may not fully represent real-world chart generation complexity. Future work should explore datasets with human-authored descriptions and real-world data.
- Model dependency: The framework relies on a proprietary model (GPT-40-mini) and three open-source models. Exploring a wider range of LLMs could reveal more about model-specific strengths and weaknesses.
- Simple agent setup: The current agentic framework is a basic two-agent setup without memory, planning, or retrieval modules. More advanced agent designs could lead to better performance in complex scenarios.
Future work should focus on:
- Developing benchmarks that better reflect real-world use cases, emphasizing visual clarity, semantic accuracy, and accessibility.
- Enhancing chart readability, aesthetics, and inclusivity, particularly for users with visual or cognitive impairments.
- Exploring more sophisticated agentic systems that incorporate memory, planning, and richer tool use.
Summary updated