Demonstration of VegaPlus: Optimizing Declarative Visualization Languages

Junran Yang, Hyekang Kevin Joo, Sai S. Yerramreddy, Siyao Li, Dominik Moritz, Leilani Battle

TL;DR

VegaPlus is a middleware system that brings big-data scalability to declarative visualization languages like Vega. By automatically partitioning data processing tasks between the browser and a backend database, it allows designers to build highly interactive visualizations for datasets with tens of millions of records without needing to write a single line of server-side code.

What Problem Does This Paper Address?

Declarative visualization languages (like Vega and D3) have revolutionized how we design interactive charts, but they hit a performance wall when dealing with "big data." Because these languages typically process and render data entirely within the browser (client-side), they struggle once a dataset exceeds a few hundred thousand rows.

Currently, if a developer wants to visualize millions of records, they usually have to manually build a custom backend, write complex SQL queries, and manage the communication between the client and server. This requires expertise in both UI design and database management. Existing "big data" visualization tools often solve this by being highly specialized (e.g., only for maps), sacrificing the creative flexibility that makes Vega so popular. VegaPlus aims to provide the best of both worlds: the expressiveness of Vega with the power of a professional database.

Methods

The authors introduce a middleware layer that acts as an intelligent bridge between the visualization specification and the data. The core workflow involves four key steps:

  1. SQL Rewriting: VegaPlus analyzes the JSON-based Vega specification and automatically translates data transforms (like filters, bins, and aggregations) into optimized SQL queries.
  2. Automated Partitioning: The system decides where to "cut" the dataflow graph. It evaluates which operations should stay on the server (to leverage the DBMS's speed and reduce data transfer) and which should happen on the client (to keep interactions snappy).
  3. Server Query Optimization: To minimize network latency, the middleware merges multiple individual operations into single, complex SQL statements, reducing the number of "round trips" between the browser and the database.
  4. Interaction-Aware Prefetching: Using user modeling techniques, the system predicts likely future interactions (like a user moving a slider) and proactively fetches that data during idle time, caching it for near-instant response.

Datasets

The authors demonstrate VegaPlus using two primary real-world datasets:

  • US Airline Flights (1987–2008): A massive dataset containing arrival and departure details for all commercial flights in the US. This is used to demonstrate high-volume histograms and record counting.
  • Census-based Occupation History (1850–2000): A longitudinal dataset used to showcase complex stacked area graphs and multi-parameter filtering (e.g., by gender and job type).

The system currently supports backends including PostgreSQL, OmniSciDB, and DuckDB.

Key Insights and Ideas

  • The "Sweet Spot" for Offloading: The researchers found a performance threshold. For datasets under 4 million rows, standard client-side Vega is often faster due to the lack of network overhead. However, for datasets exceeding 10 million rows, the computational benefits of a DBMS far outweigh the network costs, making VegaPlus significantly faster.
  • Dataflow as a Partitioning Problem: By treating a visualization as a directed graph of data transformations, the team realized they could treat optimization as a graph-cutting problem. This allows the system to remain "agnostic" of the specific database used.
  • Live "Performance Dashboard": A unique feature of the VegaPlus demo is that it doesn't just hide the complexity; it exposes it. Users can see the execution plan, see which nodes are on the server vs. client, and even manually toggle nodes to see how it affects latency in real-time.

Key Results

  • Scalability: VegaPlus successfully handles datasets with 10M+ records, where standard Vega would typically hang or crash the browser.
  • Latency Reduction: By merging SQL queries and pushing derived conditions (like filters) down into the database, the system minimizes the amount of data sent over the wire, which is usually the primary bottleneck in web-based visualization.
  • User Empowerment: The system allows "live programming" of big data visualizations. A user can modify a Vega spec in a text editor and see the optimized, database-backed result rendered live without manual backend reconfiguration.

Limitations and Future Work

While the system is highly automated, the authors note that the "optimal" split between client and server can change based on fluctuating network latencies. Future work could involve more dynamic, real-time re-partitioning as network conditions change. Additionally, while the system supports many common Vega transforms, expanding support to the entire library of complex statistical transforms remains an ongoing engineering challenge. The authors also suggest that more sophisticated user-intent models could further improve the accuracy of their prefetching algorithms.

Summary updated

Abstract

While many visualization specification languages are user-friendly, they tend to have one critical drawback: they are designed for small data on the client-side and, as a result, perform poorly at scale. We propose a system that takes declarative visualization specifications as input and automatically optimizes the resulting visualization execution plans by offloading computational-intensive operations to a separate database management system (DBMS). Our demo emphasizes live programming of visualizations over big data, enabling users to write or import Vega specifications, view the optimized plans from our system, and even modify these plans and compare their performance via a dedicated performance dashboard.
BibTeX
@inproceedings{yang2022demonstration,
  author = {Junran Yang and Hyekang Kevin Joo and Sai S. Yerramreddy and Siyao Li and Dominik Moritz and Leilani Battle},
  title = {{Demonstration of VegaPlus: Optimizing Declarative Visualization Languages}},
  booktitle = {SIGMOD},
  year = {2022},
  doi = {10.1145/3514221.3520168},
}

Loading related content...

Figure