VIST5: An Adaptive, Retrieval-Augmented Language Model for Visualization-oriented Dialog

Henrik Voigt, Nuno Carvalhais, Monique Meuschke, Markus Reichstein, Sina Zarrieß, Kai LawonnEMNLP 2023

TL;DR

This paper introduces VIST5, a novel language model designed to make data visualization more accessible by allowing users to interact with data through natural language dialog. VIST5 leverages a retrieval-augmented approach to adapt quickly to different visualization libraries and domains, enabling users to create and modify visualizations without extensive coding knowledge.

What Problem Does This Paper Address?

Interacting with data visualization tools often requires specialized knowledge of programming languages and visualization libraries. While natural language interfaces for data visualization (V-NLI) have emerged, they typically rely on tailored dialog agents and domain-specific tools, making them difficult to transfer across different applications. This creates a significant barrier for researchers and professionals who need to explore data from various domains using diverse visualization tools. The challenge is to create a V-NLI system that is easily adaptable to new domains and can seamlessly integrate with different visualization libraries without requiring extensive retraining or new code development for each application.

Methods

VIST5 employs a retrieval-augmented T5 language model architecture. The core idea is to leverage a pre-trained T5 model and enhance it with a retrieval mechanism. This retrieval component accesses a "long-term memory" containing domain-specific examples of user queries and their corresponding visualization actions. When a user submits a query, VIST5 first retrieves relevant examples from this memory. These examples, along with the current visualization state, table information, and dialog history, are then used to construct a prompt for the T5 model. The model, trained on a mixture of visualization-specific tasks and general text-to-text translation, predicts an action (e.g., creating a chart, changing a map type) and its arguments. A dialog management component validates and executes these actions, updating the visualization accordingly. This retrieval-augmented approach allows for modular extensibility and rapid adaptation to new domains and functionalities using a few-shot learning paradigm, meaning the system can learn from a small number of examples.

Datasets

VIST5 is fine-tuned using the following datasets:

  • nvbench: A large-scale, synthesized dataset for the NL2VIS task, containing text queries translated into Vega-Lite JSON specifications. It includes 25,750 examples across 750 data tables and 105 domains.
  • NIv2: A natural instructions dataset used for few-shot instruction fine-tuning.
  • Domain-Specific Dialogs: A small dataset of 300 dialog turns collected from researchers exploring the system, specifically focusing on climate science data.

Key Insights and Ideas

  • Retrieval Augmentation for Adaptability: The core innovation is using retrieval-augmented language models to adapt V-NLIs. Instead of retraining the entire model for each new domain or visualization library, VIST5 retrieves relevant "few-shot" examples to guide the model's responses. This makes the system highly modular and adaptable.
  • Few-Shot Learning for Rapid Integration: The system's reliance on few-shot learning allows for the integration of new custom intents and functionalities in a matter of minutes, significantly reducing development time compared to traditional methods.
  • Bridging the Gap Between NLP and Visualization: VIST5 demonstrates a practical approach to bridging the gap between natural language processing and data visualization, making complex data exploration more accessible to a wider audience.
  • Focus on Efficiency: The model is designed with a focus on small model size, fast trainability, and fast inference on commodity hardware, making it suitable for applications with resource constraints.

Key Results

The user study with 24 participants revealed that VIST5 provides high-quality responses.

  • NL2VIS Tasks: The average Likert score across low-level visualization tasks was 3.82, with several tasks (e.g., compute derived value, determine range) achieving very high scores (above 4). While some tasks showed higher standard deviations, the system generally performed well.
  • Few-Shot Tasks: The average rating for few-shot tasks was 3.77. Custom visualization and locomotion tasks scored particularly high (above 4), indicating the system's effectiveness in adapting to new functionalities with examples. Custom functionality tasks scored above 3, with higher variability.
  • User Engagement: Users exhibited a high average of 11.6 turns per dialog, suggesting strong engagement with the system.

Limitations and Future Work

The authors acknowledge several limitations:

  • Task Specificity: VIST5 is not a general-purpose dialog agent like ChatGPT and is limited to tasks it has been trained on or for which it receives few-shot examples.
  • Few-Shot Example Collisions: When integrating a very large number of tasks, similar few-shot examples might lead to suboptimal retrieval and reduced response quality. The authors suggest fine-tuning the sentence transformer model to improve example retrieval.
  • Visualization Library Limitations: The current model primarily generates specifications for the Vega-Lite library. Extending support to other libraries like D3.js or Observable Plot for complex specifications could be error-prone. Future work could involve developing library-independent plot specifications or methods for integrating code documentation.

Future work directions include exploring more robust methods for integrating diverse visualization libraries, developing strategies to handle similar few-shot examples more effectively, and potentially generating training data for complex specifications using larger models or human annotation.

Summary updated

BibTeX
@inproceedings{voigt2023vist,
  author = {Henrik Voigt and Nuno Carvalhais and Monique Meuschke and Markus Reichstein and Sina Zarrieß and Kai Lawonn},
  title = {{VIST5: An Adaptive, Retrieval-Augmented Language Model for Visualization-oriented Dialog}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure