Applying Supervised Learning to Real-World Problems

AAAI 1999

TL;DR

This paper argues that applying supervised learning to real-world problems often requires significant modifications to standard algorithms. It proposes a thesis focused on developing general-purpose techniques to handle challenges like misclassification costs, non-independent data, and varying example granularity, aiming to bridge the gap between theoretical models and practical applications.

What Problem Does This Paper Address?

The paper addresses the significant gap between the theoretical elegance of many supervised learning algorithms and their practical application to complex, real-world problems. While machine learning has seen widespread adoption in areas like speech processing, document classification, and fraud detection, these applications often don't fit the standard assumptions of i.i.d. (independent and identically distributed) data and uniform misclassification costs. Applying existing algorithms directly can lead to suboptimal performance, necessitating custom solutions. The authors' work aims to identify common procedural steps and develop robust, general-purpose techniques to overcome these challenges, reducing the need for ad-hoc solutions in each new application.

Methods

The paper outlines a thesis that will explore several key areas:

  • Learning with Misclassification Costs: Investigating methods to incorporate asymmetric loss functions into classifiers. For k>2 classes, simply altering priors based on labels is insufficient. The authors have studied methods for setting priors in decision trees to approximate arbitrary loss matrices and are exploring alternative approaches for decision trees, neural networks, and other algorithms.
  • Incorporating Prior Knowledge and Handling Non-Independent Data: Addressing how to leverage common sense knowledge not explicitly present in the dataset. It also tackles the violation of the i.i.d. assumption, noting that ensemble methods like AdaBoost and Bagging may not perform as expected when data is significantly non-independent.
  • Determining Example Granularity: Recognizing that the "granularity" of examples is a common issue. This involves two distinct problems: the granularity of features (e.g., window size in speech processing) and the granularity of target labels (e.g., predicting phonemes vs. stresses).
  • Examples with Multiple Labels: Developing approaches for scenarios where an example can belong to multiple classes simultaneously, moving beyond the standard assumption of single labels.

Datasets

Not discussed in the paper. The paper mentions "a manufacturing dataset" in the context of granularity experiments, but no specific details are provided.

Key Insights and Ideas

  • Real-world problems are messy: Standard machine learning assumptions (i.i.d. data, uniform costs) are often violated in practice, requiring significant algorithm adaptation.
  • Unified approaches are needed: Instead of ad-hoc solutions for each problem, there's a need for general-purpose techniques that can be applied across various real-world learning tasks.
  • Misclassification costs matter: Minimizing raw error rates isn't always the goal; understanding and incorporating asymmetric costs is crucial for many applications.
  • Data dependencies are a challenge: Non-independent data can undermine the effectiveness of even powerful ensemble methods.
  • Granularity is multi-faceted: Both feature representation and the definition of the target output (label granularity) are critical considerations.
  • Multi-label learning is an important extension: Many real-world scenarios involve examples belonging to multiple categories, a problem not well-addressed by standard single-label algorithms.

Key Results

Not discussed in the paper. This appears to be an abstract or proposal for a thesis, outlining the research questions and planned work rather than presenting completed results.

Limitations and Future Work

The paper itself is a proposal for future work, so its limitations are inherent to its nature as a research plan. The authors acknowledge that their work on granularity has so far only included "some experiments on a manufacturing dataset." They explicitly state their thesis will "emphasize on" the topics described, indicating these are areas for future investigation. The future work is essentially the research outlined in the "Methods" section, including developing robust techniques for misclassification costs, handling non-independent data, determining granularity, and addressing multi-label problems.

Summary updated

BibTeX
@inproceedings{1999applying,
  title = {{Applying Supervised Learning to Real-World Problems}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1999},
}
Figure