Quantification learning with deep neural networks

Lei Qi2021 · DOI

Abstract

Quantification learning—the task of estimating the class distribution of a population rather than labeling individual instances—is a critical requirement in fields ranging from epidemiology to political science. Despite its importance, the field has often relied on the "Classify and Count" (CC) approach, which is prone to systematic bias when classifiers are imperfect. This paper presents the Deep Quantification Network (DQN), the first end-to-end deep learning framework for quantification. By utilizing a tuplet-based training strategy and context-aware data augmentation, DQN achieves significant error reductions across text and image datasets, demonstrating particular robustness in scenarios with limited labeled data.

Introduction

In traditional supervised learning, the primary goal is classification: assigning a label to a single instance. However, many real-world applications require quantification: estimating the prevalence of classes within a group. For example, a public health official may not need to know which specific social media posts indicate a disease outbreak, but rather what percentage of the total population is discussing symptoms.

Quantification is distinct from classification because it does not require training and test data to be independent and identically distributed (i.e., it must handle "prior probability shift"). The common belief that quantification is a trivial byproduct of classification—simply counting the outputs of a classifier—is often incorrect. Systematic errors in a classifier, even small ones, can lead to significant biases in aggregate estimates. This research addresses these challenges by proposing a dedicated deep learning architecture for distribution estimation.

Methodology

Context-Aware Data Augmentation (Context-Aware-AUG)

To improve the fairness of underlying classifiers, the author introduces Context-Aware-AUG. Standard quantification methods often struggle when training data is imbalanced. This method generates synthetic training data for minority classes by identifying "anchor" words (nouns, verbs, and adjectives) and replacing them with synonyms or contextually similar terms. By leveraging three external knowledge sources—WordNet, word2vec, and ConceptNet—the method ensures that the generated documents remain human-readable and semantically coherent, providing a more robust foundation for the quantification pipeline.

The Deep Quantification Network (DQN)

The core contribution is the Deep Quantification Network (DQN), an end-to-end framework that jointly learns feature representations and class distribution estimates.

  • Tuplet-Based Learning: Unlike standard models that process single instances, DQN is trained on "tuplets"—sets of $m$ instances. This allows the model to learn the statistical "signature" of a distribution directly.
  • Zipf Distribution Training: To ensure the model generalizes to various class ratios, training tuplets are generated using a Zipf distribution. This forces the network to encounter and learn from highly skewed populations, mimicking real-world shifts in prevalence.
  • Loss Function: DQN formulates quantification as a maximum likelihood problem, utilizing Jensen-Shannon Divergence (JSD) to minimize the distance between the predicted class ratios and the ground truth ratios of the tuplet.

Experimental Evaluation

The effectiveness of DQN was evaluated across a diverse range of benchmarks, including:

  • General Text: Sentiment analysis (IMDB, Yelp) and topic classification (AG-NEWS, 20 Newsgroups).
  • Political Science: A dataset of 23,985 tweets from US state legislators and Congressional speeches.
  • Image Data: Nature images (Dog vs. Cat), medical imaging (Chest X-Ray, Covid-19), and agricultural pathology (leaf diseases in Apple, Corn, and Tomato plants).

Key Results

  1. Accuracy Gains: DQN variants consistently outperformed existing baselines such as CC, Adjusted Classify and Count (ACC), and domain-specific tools like ReadMe. DQN reduced the Mean Absolute Error (MAE) by 35% on general text and 55% on political documents.
  2. Cross-Domain Success: On image datasets, DQN reduced error by 38.34% compared to the best existing methods, proving that the architecture is not limited to text.
  3. Data Efficiency: DQN proved remarkably robust to small training sets. It achieved results comparable to traditional methods while using 40-50% less labeled data. This is particularly valuable for researchers facing high costs for manual human coding.

Discussion and Future Work

A key insight of this research is that a perfect quantifier does not strictly require a perfect classifier. As long as the false positives and false negatives for a specific class are balanced, the aggregate count remains accurate. DQN exploits this by optimizing for the aggregate ratio rather than individual labels.

While DQN represents a significant leap forward, challenges remain regarding hyperparameter sensitivity, specifically the choice of tuplet size ($m$). The research suggests a rule of thumb where $m$ should be approximately ten times the number of classes. Future work will focus on improving the interpretability of these "black box" models and exploring weakly supervised quantification, where models are trained only on aggregate group labels rather than individual instance labels.

Conclusion

This dissertation establishes DQN as a powerful, end-to-end solution for the quantification problem. By moving beyond the "classify and count" paradigm and embracing set-based learning, DQN provides a more accurate and data-efficient means of tracking population-level trends across various scientific and commercial domains.

Summary updated

Abstract

Quantification learning has been explored in several disciplines, resulting in different terminologies such as prior probability shift, prevalence estimation, or class ratio estimation. In many subject domains, it is important to estimate and track prevalence over time, for instance, tracking of the prevalence of diseases, customer complaint issues, trending topics in customer demands, support for political candidates in blog posts, and popular policy areas of interest to federal and state governments. Quantification learning is a task that “Given a labeled training set, induce a quantifier that takes an unlabeled test set as input and returns its best estimate of the class distribution.” However, quantification learning has received relatively far less attention compared to classification learning in computer science. The main reason is because of the mistaken belief that quantification is easily solved using a straightforward approach by classifying individual documents and calculating the ratio of instances in each predicted class to the total instances. Although both quantification and classification are in the family of supervised learning, there is a significant difference between the two learning tasks. Quantification learning does not require training data and test data to be independent and identically distributed. Unlike a classifier, a quantifier makes a prediction for a group of instances, not a single instance. The main contributions of the dissertation are summarized as follows. 1) We propose a new data augmentation method called Context-Aware-AUG to reduce quantification errors of classification-based quantification learning methods. A fair classifier gives a better estimate of class ratios. To obtain a fair classifier, Context-Aware-AUG generates additional training data for the classes with much fewer instances. The generated text documents are readable by humans, which is achieved by keeping the order of the words as in the original documents and replacing important words with their most similar words using knowledge from three external data sources. 2) We address an under-explored problem of quantification learning by proposing the first end-to-end Deep Quantification Neural Network (DQN) framework. DQN jointly learns effective feature representations and class distribution estimates. We formulate the quantification learning problem as a maximum likelihood problem. DQN can be seen as learning a mapping function of the input, a set of instances, and the output is the class ratios. We introduce two strategies to select a set of instances (termed a tuplet) for training to investigate how well the induced quantifier generalizes to test sets with class distributions different from that of the training set. We present the effectiveness of DQN by evaluating quantification tasks of text documents on four public datasets with 2, 4, and 20 classes. We performed a sensitivity analysis of DQN performance by varying tuplet sizes and training dataset sizes. Compared to that of classification-based quantification learning, DQN performance is less impacted by the size of the training dataset. In other words, DQN is a promising method when the manual labeling budget is limited. 3) We evaluate whether DQN is also effective for binary quantification and multi-class quantification for other types of data such as political text documents and image data, ranging from nature images, medical images, animal images, and agricultural images. DQN achieves the best results on almost all datasets. Compared to the best existing method in our study, DQN reduces the mean absolute error by 55% on two datasets of political documents and 38.34% on six image datasets. We found that transfer learning also helps improving performance of classification-based quantification methods.
BibTeX
@misc{qi2021quantification,
  author = {Lei Qi},
  title = {{Quantification learning with deep neural networks}},
  year = {2021},
  doi = {10.31274/td-20240329-612},
}
Figure