Improving the Accuracy of Learning Example Weights for Imbalance Classification

Yuqi Liu, Bin Cao, Jing FanICLR 2022

Overview

This paper introduces a more precise method for automatically assigning weights to training data to resolve class imbalance. By leveraging a mathematical constraint based on the Implicit Function Theorem, the authors developed an algorithm that learns exactly how example weights influence the final model. This approach significantly outperforms previous "short-term" approximation methods on both text and image classification tasks.

The Challenge of Class Imbalance

In real-world applications, data is rarely perfectly distributed. Whether building a spam filter or a medical diagnostic tool, you likely have thousands of "normal" examples but only a handful of "spam" or "disease" cases. This is known as class imbalance. Standard machine learning models often take the path of least resistance—achieving high overall accuracy by simply ignoring the minority class and predicting the majority class every time.

To counter this, researchers use example weighting, which gives rare examples more "importance" during training. While weights can be set manually (e.g., based on inverse frequency), the best results typically come from meta-learning, where a secondary algorithm learns the optimal weights by evaluating performance on a small, balanced validation set.

However, current meta-learning methods have a critical flaw: they rely on local approximations. They estimate how a tiny change in weight affects the model in the very next training step. This is akin to navigating a mountain range by only looking at the ground directly in front of your toes—it is inaccurate and fails to capture the long-term relationship between the weights and the final, fully optimized model.

A New Methodology: Learning with a Constraint

The authors propose a mechanism to bridge this gap by treating the model's optimality as a mathematical constraint.

  1. The Precise Relationship: Instead of guessing how weights affect the immediate next step, the authors use the Implicit Function Theorem to derive a precise derivative of the model parameters with respect to the weights. This calculates exactly how the final model will change if a specific example's weight is adjusted.
  2. Hessian Approximation: Calculating this precise relationship normally requires a "Hessian matrix" (second-order derivatives), which is computationally massive for deep neural networks. To maintain efficiency, the authors use a diagonal approximation, calculating only the most essential parts of the matrix.
  3. The Combined Algorithm: The authors recognized that while their method ensures weights are optimal for a specific model state, it doesn't necessarily drive the model to its global optimum. To solve this, they combined their approach with an existing iterative method. The two methods work in tandem: one pushes the model toward the best general solution, while the other fine-tunes the weights with surgical precision.

Key Insights

  • Moving Beyond "Short-Termism": The primary breakthrough is identifying that local, step-by-step approximations are the bottleneck in meta-weighting. By looking at the "equilibrium" state of the model via constraints, the learned weights become much more stable and effective.
  • Second-Order Information Matters: Even a simplified (diagonal) version of the Hessian matrix provides enough structural information to vastly improve weight accuracy compared to standard first-order gradient descent.
  • Robustness to Extreme Scenarios: The method shines brightest when the imbalance is most severe. In cases where the minority class is almost non-existent, this precise weighting prevents the model from collapsing into a "majority-only" predictor.

Experimental Results

The researchers evaluated their method across diverse domains, including text classification (SST-2, SST-5), image classification (CIFAR-10, CIFAR-100), and large-scale industrial data (APS Failure dataset). They tested imbalance ratios ranging from 1:10 (mild) to 1:100 (extreme).

The proposed method consistently outperformed state-of-the-art (SOTA) techniques, particularly in extreme imbalance scenarios:

Test accuracy on SST-2 across varying imbalance ratios
  • Text Performance: On the SST-2 dataset with a severe 1:100 imbalance ratio (shown as 10:1000 above), the method achieved 80.62% accuracy, compared to just 66.68% for the previous best meta-learning approach.
Test accuracy on CIFAR-100 across varying imbalance ratios
  • Image Performance: On CIFAR-100 with a 1:100 ratio (shown as 4:400 above), the method reached 77.20%, beating the SOTA by over 5 percentage points.
  • Consistency: Across all tests, the new method proved robust across different types of neural networks, including BERT for text and All-CNN-C for images.

Limitations and Future Work

While highly effective, the approach has a few limitations:

  • Computational Overhead: Although the diagonal Hessian approximation is efficient, it remains more complex than simple first-order methods. Scaling this to truly massive models (like modern LLMs) might require further optimizations.
  • Validation Set Dependency: Like most meta-learning weighting schemes, this method requires a small, balanced validation set to guide the weight learning. In extremely rare domains, gathering even 10-20 balanced examples can be difficult.

For future work, the authors suggest exploring how this precise weighting mechanism could be applied to data augmentation, helping models decide not just which examples are important, but which synthetic variations of data yield the highest performance gains.

Summary updated

BibTeX
@inproceedings{liu2022improving,
  author = {Yuqi Liu and Bin Cao and Jing Fan},
  title = {{Improving the Accuracy of Learning Example Weights for Imbalance Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
}
Figure