TL;DR
This paper introduces the "Plotting Agent," an interactive system that allows users to manipulate data visualizations using natural language commands. By framing plot manipulation as a slot-based, task-oriented dialogue problem, the system translates natural language instructions into specific plot updates, making data visualization more accessible.
What Problem Does This Paper Address?
Data visualization is crucial for deriving insights from raw data, but creating and refining plots can be time-consuming, even with powerful libraries like Matplotlib. Existing natural language interfaces (NLIs) for plotting often focus on describing data or mathematical functions, leaving the fine-grained control of plot aesthetics and details to manual coding. This paper addresses the challenge of enabling users to directly manipulate existing plots using natural language, bridging the gap between high-level data exploration and low-level plot customization. The goal is to empower both novice users and expert programmers to iterate on visualizations more efficiently.
Methods
The Plotting Agent frames plot manipulation as a slot-based, task-oriented dialogue problem. Each "slot" represents a plot property (e.g., line color, axis scale, marker size), and the system maps natural language instructions to updates in these slots.
The core of the system is a sequence-to-sequence (seq2seq) model that takes the user's natural language request, the current plot's state (represented as slot-value pairs), and the dialog history as input. It then predicts the necessary updates to the slot-value pairs to achieve the desired plot modification.
A key innovation is the feedback mode. Since the seq2seq model can make errors, the system presents the user with a top-k list of potential plot updates. The user can then select the correct one, providing a mechanism for both immediate correction and potential future learning to improve the model's accuracy.
The system is implemented in Python using PyTorch, employing Bi-LSTMs for both the text encoder and decoder.
Datasets
The system is trained on the ChartDialogs dataset. This dataset consists of dialogs generated by pairs of humans, where one acts as the user giving instructions and the other as the agent responding to them. The paper mentions using a train/dev/test split of this dataset.
Key Insights and Ideas
- Natural Language for Plotting is Feasible: The paper demonstrates that natural language can be effectively used to control complex aspects of plot manipulation, going beyond simple data description.
- Slot-based Dialogue as a Paradigm: Framing plot manipulation as a slot-filling task simplifies the problem and allows for a structured approach to understanding user intent.
- The Power of Top-K Feedback: The feedback mechanism, where users select from a list of predicted plot updates, is crucial for handling model errors and improving user experience. It also lays the groundwork for continuous learning.
- Bridging High-Level and Low-Level Control: The system allows users to move fluidly between high-level commands (e.g., "plot") and low-level aesthetic adjustments (e.g., "change the line color to red").
Key Results
- Exact Match (EM) Accuracy: While the top-1 EM accuracy is 61%, it increases significantly to 78% with a top-5 prediction. This highlights the effectiveness of the feedback mechanism.
- Runtime: Response times are fast, averaging 0.3 seconds on modest hardware, making the interaction feel responsive and more efficient than traditional web search for solutions.
- Demonstration Capabilities: The system supports various interaction modes: uploading custom data, using synthetic data, interactive mode, prior dialog mode (replaying existing conversations), and batch mode (executing instructions from a file).
Limitations and Future Work
- Slot-based Representation Limitations: The current system is limited by the predefined slots available in the plotting library's API. Complex plot manipulations that cannot be easily formulated as slot-value pairs (e.g., "shift the legend to not obscure important parts of the plot") are not supported.
- Expressive Representations: The authors acknowledge that more expressive representations beyond simple slot-value pairs are needed to handle a wider range of plot manipulations.
- Personalization: While a basic notion of personalization is included, future work aims to adapt the system more deeply to individual user preferences and contexts.
- Expanding Functionality: Future research will focus on improving language understanding, expanding functionality to more plot components, and supporting additional plotting libraries.
Summary updated