Scalable Vega: Optimizing Declarative Visualization Languages

Junran Yang, Hyekang Kevin Joo, Sai S. Yerramreddy, Siyao Li, Dominik Moritz, Leilani Battle

Abstract

While many visualization specification languages are user-friendly, they tend to have one critical drawback: they are designed for small data on the client-side and, as a result, perform poorly at scale. We propose VegaPlus, a system that takes declarative visualization specifications as input and automatically optimizes the resulting visualization execution plans by offloading computational-intensive operations to a separate database management system (DBMS). Our demo emphasizes live programming of visualizations over big data, enabling users to write or import Vega specifications, view the optimized plans from our system, and even modify these plans and compare their performance via a dedicated performance dashboard.

Introduction

Declarative visualization languages like Vega and D3 have revolutionized how we design interactive charts. However, they hit a performance wall when dealing with "big data." Because these languages typically process and render data entirely within the browser (client-side), they struggle once a dataset exceeds a few hundred thousand rows. Currently, if a developer wants to visualize millions of records, they usually have to manually build a custom backend, write complex SQL queries, and manage the communication between the client and server—a task requiring expertise in both UI design and database management.

VegaPlus acts as a middleware system that brings big-data scalability to these languages. By automatically partitioning data processing tasks between the browser and a backend database, it allows designers to build highly interactive visualizations for datasets with tens of millions of records without needing to write a single line of server-side code.

System Architecture and Methods

The core of VegaPlus is a middleware layer that acts as an intelligent bridge between the visualization specification and the data. The system treats a visualization as a directed graph of data transformations, allowing it to treat optimization as a graph-cutting problem.

The VegaPlus system architecture, illustrating the flow from the Demo UI through the middleware components (SQL Rewriting, Partitioning, Prediction) to the backend Database.

The workflow involves four key steps:

  1. SQL Rewriting: VegaPlus analyzes the JSON-based Vega specification and automatically translates data transforms (like filters, bins, and aggregations) into optimized SQL queries.
  2. Automated Partitioning: The system evaluates which operations should stay on the server to leverage the DBMS's speed and which should happen on the client to keep interactions snappy.
  3. Server Query Optimization: To minimize network latency, the middleware merges multiple individual operations into single, complex SQL statements, reducing the number of "round trips."
  4. Interaction-Aware Prefetching: Using user modeling, the system predicts likely future interactions (e.g., moving a slider) and proactively fetches that data during idle time.

Implementation and Datasets

The system currently supports various backends, including PostgreSQL, OmniSciDB, and DuckDB. To demonstrate its capabilities, the authors utilized two primary real-world datasets:

  • US Airline Flights (1987–2008): A massive dataset containing arrival and departure details for all commercial flights in the US, used to demonstrate high-volume histograms.
  • Census-based Occupation History (1850–2000): A longitudinal dataset used to showcase complex stacked area graphs and multi-parameter filtering.

Key Insights and Results

Research indicates a "sweet spot" for offloading. For datasets under 4 million rows, standard client-side Vega is often faster due to the lack of network overhead. However, for datasets exceeding 10 million rows, the computational benefits of a DBMS far outweigh the network costs, making VegaPlus significantly faster.

The VegaPlus Demo UI, featuring a Specification Editor for live programming, a Visualization Interface for interactive charts, and a Data Viewer for inspecting results.

Key results include:

  • Scalability: Successful handling of datasets with 10M+ records where standard Vega would typically crash the browser.
  • Latency Reduction: By pushing derived conditions (like filters) down into the database, the system minimizes the data sent over the wire.
  • User Empowerment: The "live programming" interface allows users to modify a Vega spec and see the optimized, database-backed result rendered live without manual backend reconfiguration.

Limitations and Future Work

While highly automated, the optimal split between client and server can change based on fluctuating network latencies. Future work will explore dynamic, real-time re-partitioning. Additionally, while the system supports many common Vega transforms, expanding support to the entire library of complex statistical transforms remains an ongoing challenge. The authors also suggest that more sophisticated user-intent models could further improve prefetching accuracy.

Summary updated

Abstract

While many visualization specification languages are user-friendly, they tend to have one critical drawback: they are designed for small data on the client-side and, as a result, perform poorly at scale. We propose a system that takes declarative visualization specifications as input and automatically optimizes the resulting visualization execution plans by offloading computational-intensive operations to a separate database management system (DBMS). Our demo emphasizes live programming of visualizations over big data, enabling users to write or import Vega specifications, view the optimized plans from our system, and even modify these plans and compare their performance via a dedicated performance dashboard.
BibTeX
@inproceedings{yang2022scalable,
  author = {Junran Yang and Hyekang Kevin Joo and Sai S. Yerramreddy and Siyao Li and Dominik Moritz and Leilani Battle},
  title = {{Scalable Vega: Optimizing Declarative Visualization Languages}},
  booktitle = {SIGMOD},
  year = {2022},
  doi = {10.1145/3514221.3520168},
}
Figure