Generative AI for Visualization: State of the Art and Future Directions

Yilin Ye, Jianing Hao, Yihan Hou, Zhan Wang, Shishi Xiao, Yuyu Luo, Wei Zeng

Abstract

Generative AI (GenAI) has witnessed remarkable progress in recent years, demonstrating impressive performance in various generation tasks across domains like computer vision and computational design. This paper surveys the integration of GenAI into visualization frameworks (GenAI4VIS), leveraging superior generative capacity for diverse operations. We categorize GenAI methods—including sequence, tabular, spatial, and graph generation—across four major visualization stages: data enhancement, visual mapping, stylization, and interaction. By summarizing current algorithms, applications, and limitations, this survey provides insights into the challenges and research opportunities in evaluation, datasets, and the gap between end-to-end models and generative algorithms.

Introduction: A New Frontier for Data Storytelling

Traditional data visualization has long been a "high-floor" activity, requiring mastery of complex grammars like Vega-Lite or D3.js. While automated visualization has been a research goal for decades, early systems often struggled with context and messy, real-world data. The emergence of Generative AI (GenAI) offers a paradigm shift, moving beyond rigid, rule-based software toward flexible, context-aware systems.

However, applying GenAI to visualization introduces a unique challenge: data integrity. Unlike artistic image generation, a visualization must be a faithful representation of underlying numbers. This paper explores how to leverage GenAI's creative power without sacrificing truth, mapping the current landscape of GenAI4VIS.

The GenAI4VIS Framework

The authors organize the GenAI4VIS landscape by mapping generative methods to specific stages of the visualization pipeline.

Conceptual framework of GenAI4VIS, illustrating the integration of generative methods across the visualization pipeline.

Categorization by Data Structure and Method

The framework identifies four primary types of generative methods:

  • Sequence Generation: Employs RNNs and Transformers to generate visualization code or interaction sequences.
  • Tabular Generation: Uses GANs to create synthetic data for privacy protection or data augmentation.
  • Spatial Generation: Leverages VAEs and Diffusion models for 2D image synthesis, style transfer, and 3D volumetric rendering.
  • Graph Generation: Utilizes Graph Neural Networks (GNNs) to optimize layouts and infer relationships in complex networks.

Mapping Tasks to Generative Methods

The relationship between visualization tasks and these methods is complex. The following diagram illustrates how specific data types are handled by different generative approaches to fulfill core visualization tasks.

Mapping of visualization tasks to generative methods and specific data types.

Key Pipeline Stages

The survey breaks down the visualization process into four stages where GenAI is currently active:

  1. Data Enhancement: This involves inferring missing values or embedding metadata into images via steganography.
  2. Visual Mapping: Translating natural language queries into functional code (NL2VIS), effectively bridging the gap between human intent and technical execution.
  3. Stylization: Transferring aesthetic styles from artwork to charts or generating "data-driven" infographics that maintain visual appeal without losing data accuracy.
  4. Interaction: Enabling "Chart Question Answering" (CQA), where users can interact with their data through natural language dialogue.

Key Insights and Results

  • The Hybrid Approach: Most successful GenAI4VIS tools are currently "hybrid." They use GenAI for "fuzzy" tasks—like understanding intent—but rely on traditional rule-based engines for rendering to ensure data points remain accurate.
  • LLMs as Orchestrators: Systems like LIDA demonstrate that LLMs can act as end-to-end agents, summarizing data, generating goals, and writing code in a single workflow.
  • Performance Gains: Modern Transformer-based models (e.g., ncNet) significantly outperform older RNN-based approaches in NL2VIS tasks. Furthermore, Vision-Language Models (VLMs) are moving toward "OCR-free" understanding, perceiving charts holistically rather than just reading text.

Challenges and Future Directions

Despite rapid progress, several "growing pains" remain:

  • The Evaluation Gap: Standard AI metrics (like image similarity) are insufficient for visualization. New metrics are needed to measure "data faithfulness" and "analytic utility."
  • Hallucination: LLMs can confidently describe non-existent trends. Solving this "truthfulness" problem is critical for high-stakes applications in finance or medicine.
  • Dataset Diversity: Current training sets often focus on simple bar and line charts. There is a need for authentic datasets featuring complex, multi-view dashboards.
  • Explainability: There is a tension between "black-box" image generation and "explainable" code generation. The field currently favors the latter to maintain user trust and allow for manual refinement.

Summary updated

Abstract

Generative AI (GenAI) has witnessed remarkable progress in recent years and demonstrated impressive performance in various generation tasks in different domains such as computer vision and computational design. Many researchers have attempted to integrate GenAI into visualization framework, leveraging the superior generative capacity for different operations. Concurrently, recent major breakthroughs in GenAI like diffusion model and large language model have also drastically increase the potential of GenAI4VIS. From a technical perspective, this paper looks back on previous visualization studies leveraging GenAI and discusses the challenges and opportunities for future research. Specifically, we cover the applications of different types of GenAI methods including sequence, tabular, spatial and graph generation techniques for different tasks of visualization which we summarize into four major stages: data enhancement, visual mapping generation, stylization and interaction. For each specific visualization sub-task, we illustrate the typical data and concrete GenAI algorithms, aiming to provide in-depth understanding of the state-of-the-art GenAI4VIS techniques and their limitations. Furthermore, based on the survey, we discuss three major aspects of challenges and research opportunities including evaluation, dataset, and the gap between end-to-end GenAI and generative algorithms. By summarizing different generation algorithms, their current applications and limitations, this paper endeavors to provide useful insights for future GenAI4VIS research.
BibTeX
@inproceedings{ye2024generative,
  author = {Yilin Ye and Jianing Hao and Yihan Hou and Zhan Wang and Shishi Xiao and Yuyu Luo and Wei Zeng},
  title = {{Generative AI for Visualization: State of the Art and Future Directions}},
  booktitle = {Vis. Informatics},
  year = {2024},
  url = {https://openreview.net/forum?id=uMznH1klbg},
}
Figure