Keynotes

Jian Pei2022 · DOI

TL;DR

This keynote introduces the burgeoning field of data valuation within emerging data and machine learning model markets. It proposes using cooperative game theory, specifically Shapley values, as a principled method to fairly price data and models, even at scale, and explores challenges in applying these concepts to cooperative machine learning and privacy-preserving marketplaces.

What Problem Does This Paper Address?

The paper addresses the critical challenge of data valuation in the context of rapidly growing data markets and AI applications. As data becomes a valuable commodity, understanding how to price it fairly and accurately is paramount. This is especially true when multiple data owners contribute to a shared dataset or when machine learning models are developed and traded. The core problem is establishing a robust and scalable mechanism to determine the worth of individual data contributions or model components, ensuring equitable distribution of value and fostering trust in these new markets.

Methods

The keynote proposes leveraging cooperative game theory, with a particular focus on Shapley values, as a principled approach to data valuation. Shapley values, derived from game theory, offer a way to fairly distribute the total gains of a coalition among its players based on their marginal contributions. The authors suggest that by making certain practical assumptions about the utility of data products, it's feasible to assess Shapley values for millions of data products and tens of owners, making the approach scalable. The talk also touches upon the challenges of modeling and computing fair reward allocations in cooperative machine learning settings like federated learning, and the complexities of building privacy-preserving model marketplaces.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Data Valuation is Crucial for Data Markets: The emergence of data markets necessitates a robust framework for valuing data, akin to how other commodities are priced.
  • Shapley Values as a Principled Solution: Cooperative game theory, specifically Shapley values, offers a theoretically sound and fair method for attributing value to individual data contributions or model components.
  • Scalability is Achievable: Despite the potential complexity, the authors suggest that with practical assumptions, Shapley value computation can be made practical even for large-scale scenarios involving millions of data products and numerous owners.
  • Cooperative ML Presents New Challenges: Applying fair reward allocation in collaborative machine learning paradigms like federated learning introduces unique modeling and computational hurdles.
  • Privacy is a Key Consideration: Building marketplaces for models and data must inherently address privacy concerns to ensure user trust and data security.

Key Results

Not discussed in the paper. The paper is a keynote abstract and focuses on the proposed approach and challenges rather than presenting empirical results.

Limitations and Future Work

The abstract highlights the challenges in modeling and computing fair reward allocation in one-shot cooperative machine learning processes, such as federated learning. It also points to the difficulties in building privacy-preserving model marketplaces. These represent areas for future research and development, suggesting that while the proposed Shapley value approach is promising, its practical implementation in complex, privacy-sensitive, and cooperative settings requires further investigation.

Summary updated

Abstract

Data markets are emerging and promising for harvesting data from many data owners to support datadriven AI applications and many second-uses of big data. Data valuation, such as pricing, plays a central role in data markets. In this talk, I will survey the motivations and the state-of-the-art practice of data and (machine learning) model markets, and review data valuation in end-to-end data analytics and machine learning pipelines. Then, I will focus on models, fairness, and scalability of data valuation using some well-established solution concepts in cooperative game theory, such as Shapley value. As a principled approach, I will illustrate that with some simple yet practical assumptions about the utility of data products, assessing accurate Shapley value of millions of products and tens of owners is highly practical. I will also demonstrate the challenges in modeling and computing fair reward allocation in one-shot cooperative machine learning processes, such as federated learning, as well as in building privacy preserving model marketplaces.
BibTeX
@misc{pei2022keynotes,
  author = {Jian Pei},
  title = {{Keynotes}},
  year = {2022},
  doi = {10.1109/cogmi56440.2022.00009},
}
Figure