Try Before You Buy: A practical data purchasing algorithm for real-world data marketplaces

Santiago Andrés Azcoitia, Nikolaos Laoutaris

TL;DR

This paper introduces "Try Before You Buy" (TBYB), a novel algorithm for data buyers in marketplaces that significantly improves purchasing efficiency. TBYB allows buyers to achieve near-optimal results with only a fraction of the information required by traditional methods, making data acquisition more practical and profitable.

What Problem Does This Paper Address?

Data marketplaces are booming, but buying data is tricky. Unlike physical goods, digital data can be copied infinitely, making its value hard to determine before purchase. Buyers need to know how well a dataset will perform with their specific AI/ML algorithms, but this information is often unavailable or prohibitively expensive to obtain for all possible combinations of datasets. This "chicken-and-egg" problem forces buyers to make decisions based on limited information, often leading to suboptimal purchases or even losses. The paper addresses this by proposing a practical solution that bridges the gap between theoretical full-information models and the reality of real-world data marketplaces.

Methods

The core contribution is the "Try Before You Buy" (TBYB) algorithm. TBYB operates sequentially and greedily, leveraging information about the accuracy of individual datasets with the buyer's AI/ML task, rather than requiring knowledge of all possible dataset combinations. Two versions are presented:

  • S-TBYB (Stand-alone): This version assumes the marketplace provides individual dataset accuracies. The algorithm iteratively selects datasets based on their expected profit, with a risk parameter to bound potential losses.
  • A-TBYB (Assisted): This version allows buyers to query the marketplace for the marginal accuracy of datasets given the data they already own. This provides more refined information for decision-making.

The paper also contrasts TBYB with value-unaware heuristics (volume-based, price-based) and an optimal full-information strategy.

Datasets

The paper uses both synthetic data and real-world spatio-temporal data for evaluation.

  • Synthetic Data: A synthetic model is developed to control parameters like total cost of data (TCOD), marginal utility profile (MUP), and data interchangeability (DI). This allows for systematic evaluation of the algorithms under various conditions.
  • Real-world Data: The authors use a dataset of 11.1 million taxi rides from the City of Chicago (obtained from data.cityofchicago.org) to train a multiseasonal SARIMA forecasting model. This dataset represents 15 large taxi companies and a hypothetical 16th company aggregating smaller ones, serving as data sellers.

Key Insights and Ideas

  • "Try Before You Buy" is a practical solution: TBYB offers a way to approximate the efficiency of full-information models without requiring all that information, making it suitable for real-world marketplaces.
  • Individual dataset accuracy is powerful: Knowing how each dataset performs individually is surprisingly effective for making good purchasing decisions.
  • Risk management is crucial: The inclusion of a risk parameter (λ) in TBYB allows buyers to balance potential gains with acceptable losses, a critical aspect in uncertain data markets.
  • Value-agnostic heuristics are often suboptimal: Strategies that ignore the actual value of data for the buyer's specific task (e.g., purely volume-based or price-based) perform significantly worse, especially in complex scenarios.
  • Data interchangeability matters: TBYB's advantage over heuristics increases when datasets are less interchangeable, as knowing their specific value becomes more critical.

Key Results

  • Near-optimal performance: TBYB (both S-TBYB and A-TBYB) consistently achieves profit levels close to the optimal full-information strategy across a wide range of parameters. For instance, in many scenarios, TBYB achieves over 90% of the optimal profit.
  • Outperforms heuristics: TBYB significantly outperforms value-unaware heuristics, especially in realistic scenarios with high total data costs (TCOD), concave value functions, non-interchangeable datasets, and pricing not tied to value.
  • Robustness: TBYB remains effective even when dataset prices are not directly correlated with their actual value for the buyer.
  • Real-world validation: The taxi demand forecasting use case demonstrates that TBYB's advantages hold with real spatio-temporal data, outperforming volume-based and price-based purchasing strategies.

Limitations and Future Work

  • Synthetic model limitations: The synthetic model, while useful, simplifies how datasets mix and contribute to accuracy. Real-world interactions can be more complex.
  • Buyer value function assumption: The synthetic evaluation assumes buyer value directly maps to accuracy. In reality, buyers may have diverse value functions that translate accuracy into monetary worth differently.
  • Future Work: The authors are developing a fully functional TBYB prototype for real users. Future work will also explore dynamic data pricing, arbitrage protection, price discrimination, and engineering challenges like scalability and security for building complete data marketplaces.

Summary updated

BibTeX
@inproceedings{azcoitia2022try,
  author = {Santiago Andrés Azcoitia and Nikolaos Laoutaris},
  title = {{Try Before You Buy: A practical data purchasing algorithm for real-world data marketplaces}},
  booktitle = {DE@CoNEXT},
  year = {2022},
  doi = {10.1145/3565011.3569054},
}
Figure