Introduction
In classical statistics, small datasets are often analyzed with a high ratio of graphed values to data values, allowing for deep exploratory data analysis (EDA). However, as datasets scale to billions of observations, this ratio typically collapses, forcing analysts to rely on heavy aggregation or simple summary statistics. This paper introduces Visualization Databases (VDBs), a framework designed to maintain the depth of EDA at scale by generating thousands of coordinated plots organized into multi-page, searchable databases.
Methodology
The VDB framework relies on several key technical and design pillars:
- Partitioning and Sampling: Data is broken into "core partitions" (e.g., by user or location). Analysts apply visualization methods to these subsets, using either exhaustive or representative sampling to cover the multidimensional space.
- Trellis Displays: Utilizing a grid of panels across multiple pages ensures consistency, allowing analysts to rapidly scan through hundreds of plots to detect anomalies or patterns.
- Automation Algorithms: To manage the sheer volume of plots, algorithms like "Banking to 45 degrees" are used. This automatically adjusts plot aspect ratios so the average slope of lines is 45°, the optimal angle for human perception of rate changes.
- Distributed Computing: The authors leverage RHIPE (R + Hadoop) for embarrassingly parallel computation, making the generation of a 10,000-page VDB computationally feasible.
Case Studies
The Jester Dataset
The authors analyzed 73,421 raters providing ratings for 100 jokes. A VDB consisting of a 100-page Trellis display was used to check model residuals for 4,000 individual raters.
As shown in Figure 1, the VDB allows for the visual confirmation that a normal distribution is a suitable approximation for error marginal distribution across a vast user base.
Connection Dataset
In a study of 1.5 billion TCP packets (146 GB raw data), the VDB helped classify "interactive ssh" connections versus automated ones. A key insight was the use of juxtaposition—placing client and server data back-to-back—rather than superposition (layering).
Figure 6 illustrates how this design choice leverages Gestalt principles to make connection "signatures" immediately apparent to the human eye.
Key Insights
- The Analyst Time Paradox: The effort to write code for a display is independent of the number of subsets it is applied to. Parallel computing makes massive plot collections "cheap" to produce.
- Controlled Animation: Consistent page design turns rapid flipping through a PDF into a form of animation, where the visual system effortlessly flags departures from trends.
- Aspect Ratio as a Variable: Automated banking is essential because the physical slope of a line segment significantly impacts the accuracy of information decoding.
Conclusion
Visualization Databases prove that we need not simplify data to fit traditional displays. By expanding the infrastructure to match data complexity, analysts can scan 100-page displays in minutes to find model failures or hidden dependencies. Future work will focus on specialized VDB viewers and optimizing the balance between simultaneous and sequential visual assessment.
Summary updated