The Importance of the Test Set Size in Quantification Assessment

André Gustavo Maletzke, Waqar Hassan, Denis Moreira dos Reis, Gustavo E. A. P. A. BatistaIJCAI 2020 · DOI

TL;DR

This paper argues that the size of the test set is a critical, yet often overlooked, factor in evaluating quantification algorithms. They demonstrate that performance rankings of quantifiers can drastically change with test set size, leading to potentially misleading conclusions. To address this, they propose a meta-learning framework that recommends the best quantifier based on the test set size, outperforming individual methods.

What Problem Does This Paper Address?

Quantification is a machine learning task focused on estimating the proportion of each class within a dataset, rather than classifying individual instances. While classification research extensively studies the impact of training set size on performance, the quantification community has largely ignored the influence of test set size. This oversight has several detrimental consequences:

  1. Implicit Assumption of Uniform Performance: Researchers often assume that a quantifier performing well on large test sets will also perform well on small ones, which the paper shows is not true.
  2. Cherry-Picking Risk: The choice of test set size can be manipulated to favor a particular algorithm, leading to biased evaluations.
  3. Disregard for Method Design: The importance of designing methods that are robust across various test set sizes is neglected.

This paper aims to highlight this critical issue and propose a more robust evaluation methodology and a practical solution for selecting appropriate quantifiers.

Methods

The authors propose a comprehensive experimental setup to evaluate quantification algorithms across varying test set sizes. Their approach involves:

  • Extensive Evaluation: Comparing 12 quantification methods on 13 benchmark datasets, with 14 different test set sizes (ranging from 10 to 500 instances) and 100 class distributions.
  • Meta-Learning Framework (MLQ): To address the problem of quantifier selection based on test set size, they introduce a meta-learning approach. This framework learns to predict the best quantifier for a given test set by considering metafeatures, including the test set size itself. The MLQ framework trains a classifier where the "classes" are the different quantification algorithms.
  • Novel Experimental Design: They introduce an additional step in the standard evaluation procedure to assess performance on distinct test sizes, in addition to varied class distributions.

Datasets

The paper utilizes a variety of publicly available datasets from repositories like UCI, OpenML, and Reis. Specific datasets mentioned include:

  • AedesSex
  • Anuran Calls
  • ArabicDigit
  • BNG (vote)
  • Spambase
  • Handwritten-QG
  • Wine Type
  • Wine Quality
  • HTRU2
  • Letter Recognition
  • Mozilla4
  • Nomao
  • Occupancy Detection

The authors note that they primarily used binary quantification datasets to simplify the experimental setup and allow for a wider range of algorithms to be evaluated.

Key Insights and Ideas

  • Test Set Size is Crucial: The performance ranking of quantification algorithms can change dramatically depending on the test set size. What works well for large sets might fail on small ones, and vice versa.
  • Small Test Sets are Particularly Challenging: Current quantifiers often exhibit mediocre performance on small test sets (e.g., 10 instances). Only a few methods, like Median Sweep (MS) and SORD, manage to outperform the naive "Classify and Count" (CC) baseline in these scenarios.
  • Distribution Matching Methods Excel on Large Sets: Algorithms based on distribution matching, particularly the DyS framework, tend to perform best on larger test sets, as they benefit from more data to model distributions accurately.
  • Meta-Learning for Quantifier Selection: A meta-learning approach (MLQ) can effectively recommend the best quantifier for a given test set size, outperforming individual best-performing quantifiers for specific scenarios.
  • Literature's Oversight: The common practice of evaluating quantifiers on a single, arbitrary test set size provides an incomplete picture and can lead to inaccurate conclusions and recommendations.

Key Results

  • Performance Variability: Figure 1 (and subsequent figures) clearly shows significant shifts in algorithm rankings when comparing test sets of 500 instances versus 10 instances. For example, distribution matching methods dominate large test sets, while methods like MS and SORD perform better on small test sets.
  • MLQ Outperforms Baselines: The proposed MLQ framework, which uses meta-learning to select quantifiers based on test set size, consistently outperforms a random selection baseline (RND) and shows promise in approaching an ideal "topline" performance.
  • Mediocre Performance on Small Sets: The paper highlights that many current quantifiers struggle with small test sets, with performance often degrading significantly compared to larger sets.

Limitations and Future Work

  • Limitations:
    • The study primarily focused on binary quantification datasets, although the authors argue the findings are generalizable to multiclass scenarios.
  • While MLQ shows improvement, it does not reach the performance of an ideal "topline" scenario where the absolute best quantifier is always chosen.
  • Future Work:
    • The authors suggest that future research should focus on developing quantification methods that are robust across a wide range of test set sizes.
  • They also propose that future quantification research should adopt their proposed experimental setup for evaluation and comparison.
  • Further investigation into the impact of score quality on quantification methods is also suggested.

Summary updated

BibTeX
@inproceedings{maletzke2020importance,
  author = {André Gustavo Maletzke and Waqar Hassan and Denis Moreira dos Reis and Gustavo E. A. P. A. Batista},
  title = {{The Importance of the Test Set Size in Quantification Assessment}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/366},
}
Figure