Subset Selection in Machine Learning: Theory, Applications, and Hands On

Rishabh K Iyer, Abir De, Ganesh Ramakrishnan, Jeff BilmesOpenReview 2022 · OpenReview

TL;DR

This tutorial proposal outlines a comprehensive approach to subset selection in machine learning, bridging theoretical foundations with practical applications. It aims to equip researchers and practitioners with tools and knowledge to build more efficient, robust, fair, and personalized AI systems by intelligently selecting data subsets, features, or model components.

What Problem Does This Paper Address?

Modern deep learning models achieve remarkable performance but come with significant drawbacks: they require vast amounts of labeled data, extensive computational resources for training, and can be brittle to noisy or out-of-distribution data, often inheriting biases. This tutorial addresses the need for more efficient and responsible AI by exploring how subset selection can mitigate these issues. Instead of using all available data or features, the focus is on identifying the most informative or representative subsets to achieve desired outcomes with fewer resources and improved model properties.

Methods

The tutorial delves into the theoretical underpinnings of subset selection, including:

  • Coresets: Techniques for constructing small, representative subsets of data that approximate the properties of the full dataset, enabling faster training and analysis.
  • Submodularity: A mathematical framework that captures the diminishing returns property of adding elements to a set. This is leveraged for tasks like data summarization, feature selection, and active learning, where the value of adding more data points decreases.
  • Combinatorial Optimization: The study of finding optimal solutions from a discrete set of possibilities, which is central to many subset selection problems.

The tutorial also highlights practical applications and introduces several toolkits developed by the authors:

  • SUBMODLIB: For submodular optimization and data summarization.
  • CORDS: For coresets and subset selection for compute-efficient training.
  • DISTIL: For active learning strategies.
  • APRICOT: For submodular optimization in machine learning.
  • TRUST: For targeted subset selection for robust, fair, and personalized learning.

Datasets

Not explicitly discussed in the provided text, as this is a tutorial proposal. The focus is on the methodologies and applications rather than specific benchmark datasets.

Key Insights and Ideas

  • Subset selection is a unifying theme: Many seemingly disparate problems in ML (e.g., efficient training, active learning, feature selection, model compression, fairness) can be effectively addressed through the lens of subset selection.
  • Beyond accuracy: The tutorial emphasizes that subset selection can improve not just efficiency but also crucial aspects like robustness, fairness, and personalization, moving beyond a sole focus on model accuracy.
  • Bridging theory and practice: A key goal is to connect theoretical concepts like submodularity and coresets with their practical implementation and impact in real-world applications.
  • Hands-on experience is crucial: The inclusion of a dedicated hands-on session with practical toolkits aims to empower attendees to immediately apply these techniques.

Key Results

As this is a tutorial proposal, specific quantitative results are not presented. However, the text mentions that state-of-the-art approaches in subset selection have demonstrated significant benefits, including:

  • 5x - 10x speedups and energy reductions in training with negligible accuracy loss.
  • Reduced labeling costs through efficient selection of unlabeled data.
  • Improved robustness, fairness, and personalization in models.

Limitations and Future Work

The authors acknowledge that while subset selection has been considered in isolation for various applications, this tutorial aims to unify these efforts. They also highlight that this is the first instance of this specific tutorial, suggesting it's a new initiative to bring together different sub-communities. Future work is implicitly suggested by the breadth of applications covered and the introduction of new toolkits, encouraging further research and adoption of these methods. The tutorial aims to foster a broader discussion on topics like Green AI, fairness, and data efficiency.

Summary updated

Abstract

Machine learning, and specifically deep learning has trans- formed numerous application domains like computer vi- sion and video analytics, speech recognition, natural lan- guage processing, and so on. As a result, significant focus of researchers in the last decade has been on obtaining the most accurate models, often matching and sometimes sur- passing human level performance in these areas. However, deep learning is also unlike human learning in many ways. To achieve the human level performance, deep models re- quire large amounts of labeled training data, several GPU instances to train, and massive size models (ranging from hundreds of millions to billions of parameters). In addition, they are often not robust to noise, imbalance, and out of distribution data, and can also easily inherit the biases in the training data. Motivated by these desiderata and many more, we will present a rich framework of subset selection and coreset based approaches. We will begin by studying the theoretical aspects of subset selection, such as the modeling paradigms of coresets and submodularity, and the resulting optimization algorithms and theoretical properties. We will then study the application of subset selection to a number of areas like: a) compute-efficient training of deep models, b) label efficient methods like active learning, c) feature selec- tion and model compression, d) targeted subset selection for robust, fair, and personalized learning, and e) human assisted learning. An important component of this tutorial will be a hands-on session where we will present a number of toolk- its developed as a part of DECILE (www.decile.org), which include SUBMODLIB (submodular optimization), CORDS (coresets and subset selection for compute efficient training),
BibTeX
@misc{iyer2022subset,
  author = {Rishabh K Iyer and Abir De and Ganesh Ramakrishnan and Jeff Bilmes},
  title = {{Subset Selection in Machine Learning: Theory, Applications, and Hands On}},
  year = {2022},
  url = {https://openreview.net/forum?id=iLPXBmaE-6d},
  howpublished = {OpenReview},
}
Figure