Trust, but Verify: Optimistic Visualizations of Approximate Queries for Exploring Big Data

Dominik Moritz, Danyel Fisher, Bolin Ding, Chi Wang

Abstract

Analysts require interactive speeds for exploratory data analysis (EDA), yet modern big data systems often impose significant latency. While Approximate Query Processing (AQP) via sampling offers a path to responsiveness, it frequently compromises user trust and accuracy. We introduce optimistic visualization, a user experience framework that enables analysts to explore approximate results interactively while providing robust mechanisms for background verification and error recovery. Our prototype system, Pangloss, implements these principles. Through a laboratory study with expert visualizers and case studies at Microsoft, we demonstrate that this "trust, but verify" approach enhances analyst confidence and maintains creative flow without sacrificing precision.

The Challenge of Big Data Exploration

Exploratory Data Analysis is inherently iterative. To maintain a "flow state," analysts typically need query results in under a second. However, as datasets scale to hundreds of millions or billions of rows, traditional database engines can take minutes to return a single result, stalling the exploration process.

Approximate Query Processing (AQP) addresses this speed bottleneck by using data sampling to provide rapid "best guess" answers. However, AQP introduces a trust deficit: analysts fear that samples might miss critical outliers or introduce statistical bias. Furthermore, "progressive" visualization systems that update continuously can be distracting, as "dancing bars" fluctuate while more data is scanned.

Optimistic Visualization and Pangloss

To bridge the gap between speed and trust, we propose Optimistic Visualization. This pattern allows the system to:

  1. Show an approximation instantly using a small data sample.
  2. Allow the user to "remember" a view, triggering a full, precise query in the background.
  3. Notify the user when the precise result is ready, enabling a direct comparison between the initial estimate and the ground truth.

Pangloss, our prototype implementation, sits atop the Sample+Seek AQP engine. A core innovation in Pangloss is the concept of Distribution Uncertainty. Rather than relying solely on per-group confidence intervals (error bars), Pangloss calculates a global metric representing how much the overall shape of the visualization might deviate from the true distribution.

Figure 1: The framework of Distribution Uncertainty. (A) shows the initial approximation with error bars. (B) and (C) illustrate distributions that fall within or outside the calculated global uncertainty, respectively.

Evaluation and Real-World Application

We evaluated Pangloss using several massive real-world datasets, including 170 million rows of BTS flight data, software telemetry, search engine logs (nearly 1 billion rows), and the Twitter firehose.

Key Performance and User Results

  • Speed vs. Precision: Pangloss delivered approximate results for 170 million rows in approximately 100ms (with 1–2% uncertainty), compared to roughly one minute for a full SQL query on identical hardware.
  • Workflow Integration: Analysts treated approximations as "drafts" to debug queries. If a fast view looked incorrect, they could adjust filters immediately rather than waiting for a slow, precise query to confirm a typo.
  • The "Ship Room" Requirement: While comfortable with samples for exploration, analysts required precise data for high-stakes decision-making. Optimistic visualization supports this by providing interactive speed for the analyst and verifiable precision for the boardroom.
Figure 2: Comparison of two distributions that both fall within the calculated uncertainty bounds, illustrating how the 'true' answer might vary from the initial estimate.

Insights and Future Directions

A significant finding was the value of identifying "missing" data. Sampling often misses the "long tail"; Pangloss highlights newly discovered groups in the precise result, helping analysts understand the limitations of their initial samples.

Despite its success, Pangloss has limitations. Currently, users must manually trigger precise queries, and the system struggles with non-linear transformations like logarithmic scales. Future work will explore automated verification triggers and support for more complex visualization types beyond bar charts and heatmaps. By treating approximations as a starting point rather than the final answer, optimistic visualization provides a scalable path for interactive big data exploration.

Summary updated

Abstract

Analysts need interactive speed for exploratory analysis, but big data systems are often slow. With sampling, data systems can produce approximate answers fast enough for exploratory visualization, at the cost of accuracy and trust. We propose optimistic visualization, which approaches these issues from a user experience perspective. This method lets analysts explore approximate results interactively, and provides a way to detect and recover from errors later. Pangloss implements these ideas. We discuss design issues raised by optimistic visualization systems. We test this concept with five expert visualizers in a laboratory study and three case studies at Microsoft. Analysts reported that they felt more confident in their results, and used optimistic visualization to check that their preliminary results were correct.
BibTeX
@inproceedings{moritz2017trust,
  author = {Dominik Moritz and Danyel Fisher and Bolin Ding and Chi Wang},
  title = {{Trust, but Verify: Optimistic Visualizations of Approximate Queries for Exploring Big Data}},
  booktitle = {CHI},
  year = {2017},
  doi = {10.1145/3025453.3025456},
}
Figure