A Comparison of Models for Cost-Sensitive Active Learning

Katrin Tomanek, Udo HahnCOLING 2010

TL;DR

This paper tackles the challenge of reducing annotation costs in active learning (AL) by moving beyond simply minimizing the number of labeled examples. Instead, it proposes and evaluates three cost-sensitive AL approaches that explicitly incorporate the time it takes to annotate data, demonstrating significant savings in annotation time compared to standard methods.

What Problem Does This Paper Address?

Active Learning (AL) is a powerful technique for reducing the amount of labeled data needed to train machine learning models, thereby cutting down annotation costs. Traditionally, the efficiency of AL in Natural Language Processing (NLP) has been measured by the number of tokens or sentences saved compared to random sampling. However, this metric assumes uniform annotation costs for all data points, which is linguistically and practically questionable. The true cost of annotation, especially for complex NLP tasks, is better represented by the time it takes human annotators to label data. This paper addresses the gap between the traditional cost metric and the real-world cost of annotation by proposing and comparing methods that incorporate actual annotation time into the AL selection process.

Methods

The authors propose three cost-sensitive active learning (CoSAL) approaches:

  • Cost-Constrained Sampling (CCS): This is the simplest approach, where examples exceeding a predefined maximum cost (annotation time) are simply removed from the pool of candidates before AL selection.
  • Linear Rank Combination (LRK): This method combines utility scores (how informative an example is) and cost information by ranking both and then linearly combining these ranks. A weighting parameter a controls the trade-off between utility and cost.
  • Benefit-Cost Ratio (BCR): This approach calculates a ratio of utility (benefit) to cost. It requires a linear relationship between utility and cost, or a non-linear transformation function to align them. The authors explore using a calibrated exponential function for utility to achieve this linearity.

These CoSAL approaches are integrated into both standard fully supervised AL (FuSAL) and a semi-supervised variant (SeSAL). The utility functions explored include ULC (based on sequence confidence) and UMA (based on token-level confidence).

Datasets

The primary dataset used for evaluation is MUC7T. This is a re-annotation of a portion of the MUC7 newspaper corpus, specifically extended to include annotation time information for each entity type. This dataset contains 3,113 sentences (76,900 tokens) and is crucial for evaluating cost-sensitive methods that rely on actual annotation times.

Key Insights and Ideas

  • Annotation time is a more realistic cost metric: The paper strongly argues that simply counting tokens is inadequate and that actual annotation time is a more accurate reflection of annotation cost, especially in NLP.
  • Cost-sensitive AL can significantly reduce annotation time: By incorporating cost into the selection process, substantial savings in annotation time can be achieved compared to random selection and even standard active learning.
  • Semi-supervised AL benefits most from cost-sensitivity: The cost-sensitive semi-supervised approach (SeSAL) showed the most impressive results, reducing annotation time by up to 54% compared to random selection.
  • Utility function choice matters for cost-sensitive methods: The ULC utility function, while suitable for standard AL, proved inappropriate for cost-sensitive methods like BCR without proper calibration, highlighting the need for utility measures that reflect true benefit.
  • Early AL iterations show the most benefit: The advantages of cost-sensitive AL are most pronounced in the early stages of learning. This is attributed to corpus exhaustion effects, where the pool of examples favorable to both utility and cost becomes scarce.

Key Results

  • Cost-sensitive SeSAL reduced true annotation time by approximately 54% compared to random selection.
  • Cost-sensitive FuSAL reduced annotation time by approximately 44% compared to random selection.
  • The LRK approach with a utility function calibrated for linearity (e.g., e^β·ULC) showed promising results, outperforming cost-insensitive FuSAL.
  • BCR with a calibrated utility function also outperformed cost-insensitive FuSAL, whereas BCR with an uncalibrated ULC utility function performed poorly.
  • The paper found that cost-sensitive AL approaches tend to perform best in early AL iterations, with improvements diminishing as the corpus is exhausted.

Limitations and Future Work

  • Corpus Exhaustion: The benefits of cost-sensitive AL are most apparent in early iterations due to corpus exhaustion. This suggests that for cost-sensitive AL to be consistently effective, larger and more diverse corpora are needed to ensure a sufficient pool of examples that are favorable to both utility and cost.
  • Parameter Tuning: Some CoSAL approaches (like CCS and LRK) require corpus-specific parameter tuning (e.g., Cmax for CCS, a for LRK), which can be difficult to determine for new annotation projects. BCR, on the other hand, requires less parameterization once a calibration function is found.
  • Estimated Costs: The current experiments use actual measured annotation times. For real-world scenarios, annotation costs are often not directly known and must be estimated. The authors plan to incorporate predictive models of annotation costs (based on eye-tracking studies) into their future work.
  • Non-linear Utility Calibration: While the paper proposes a method for calibrating utility functions for BCR, finding appropriate non-linear calibration functions for other utility measures or scenarios might be challenging.

Summary updated

BibTeX
@inproceedings{tomanek2010comparison,
  author = {Katrin Tomanek and Udo Hahn},
  title = {{A Comparison of Models for Cost-Sensitive Active Learning}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2010},
}
Figure