Towards an Objective Metric for Data Value Through Relevance

Boris Glavic, Pengyuan Li, Ziyu Liu, Dieter Gawlick, Vasudha Krishnaswamy, Danica Porobic, Zhen Hua LiuCIDR 2024

Abstract

Organizations store massive amounts of data, but not all of it is equally useful. This vision paper proposes an objective metric for "data value" based on data relevance—specifically, whether a piece of data actually contributes to the result of a query, tracked via data provenance. By quantifying value this way, systems can optimize storage tiering, caching, data curation efforts, and even data pricing in marketplaces.

Introduction

As data generation explodes, storing, cleaning, and processing it consumes significant compute and human resources. Organizations must decide where to allocate these limited resources, but they currently lack an objective way to measure which data is actually valuable. Historically, systems have used data access frequency as a proxy for value. However, access does not equal relevance. For example, a database might perform a full table scan of a billion rows to evaluate a query, but if only two rows satisfy the WHERE clause, the other 999,999,998 rows contributed nothing to the final result. If we base data life-cycle decisions on access frequency, we treat all billion rows as equally "hot." This paper aims to fix this by defining data value strictly through its observable contribution to computational outputs.

Methodology: From Access to Relevance

We propose a framework that shifts the focus from data access to data relevance using data provenance (tracking which input records produced a specific output). The baseline value of a data item is defined as the fraction of queries in a workload for which that item is relevant, weighted by the frequency and potentially the monetary importance of those queries.

To illustrate this, consider a conceptual e-commerce database consisting of products and orders. In Figure 1, we see how specific product records are identified as relevant to different query workloads (indicated by color coding).

Figure 1: Products table illustrating relevance-based highlighting.

Similarly, Figure 2 shows an Orders table where provenance tracking identifies which transactions are relevant to specific analytical outputs.

Figure 2: Orders table where rows are tracked for relevance to specific outputs.

Provenance Sketches

Because tracking exact row-level provenance for every query is computationally prohibitive, we propose using provenance sketches. These are compact, coarse-grained bitvectors that track relevance at the partition or fragment level. To further reduce overhead, workloads can be approximated by clustering queries based on similar data relevance profiles and sampling them.

Key Insights and Results

  • The Illusion of "Hot" Data: Data that is frequently read but filtered out is effectively useless. True value only exists if data makes it through the interface (e.g., the SQL query) to the user.
  • Direct Monetary Translation: If an organization knows the monetary value of a query result, they can use provenance to distribute that value directly down to the specific rows that generated the answer.
  • Storage and Processing Efficiency: Tracking relevance via provenance sketches is highly compact. For a given workload, traditional materialized views might require ~44GB of storage, whereas provenance sketches require only ~200KB. Furthermore, by knowing data relevance beforehand, databases can implement Provenance-Based Data Skipping (PBDS), filtering out irrelevant data early in the pipeline to speed up processing.

Limitations and Future Work

Implementing this vision involves several challenges:

  • Overhead vs. Accuracy: There is an inherent trade-off between the granularity of provenance tracking and system overhead. Coarse sketches save space but may over-approximate relevance.
  • Evolving Workloads: Maintaining data value metrics during data updates or workload shifts requires complex incremental maintenance.
  • Dark Data: The current metric assigns zero value to unused data. Future work will integrate this with dataset discovery systems to recommend "dark data" to users, testing for unrealized potential value.

Conclusion

By grounding data value in objective relevance rather than simple access patterns, we provide a path toward more efficient and economically rational data management systems.

Summary updated

BibTeX
@inproceedings{glavic2024towards,
  author = {Boris Glavic and Pengyuan Li and Ziyu Liu and Dieter Gawlick and Vasudha Krishnaswamy and Danica Porobic and Zhen Hua Liu},
  title = {{Towards an Objective Metric for Data Value Through Relevance}},
  booktitle = {CIDR},
  year = {2024},
}
Figure