Difficulty-Net: Learning To Predict Difficulty for Long-Tailed Recognition

Saptarshi Sinha, Hiroki OhashiWACV 2023

TL;DR

This paper introduces Difficulty-Net, a novel meta-learning approach that learns to predict class difficulty for long-tailed recognition tasks. By focusing on relative class difficulty and introducing a "driver loss" function, Difficulty-Net achieves state-of-the-art performance on several challenging long-tailed datasets, outperforming existing methods that rely on fixed difficulty formulations.

What Problem Does This Paper Address?

Deep learning models excel on balanced datasets but struggle when deployed in real-world scenarios, which are often "long-tailed." This means a few classes (head classes) have many training samples, while many other classes (tail classes) have very few. This imbalance biases models towards head classes, leading to poor performance on tail classes. While weighted loss functions are a common solution, simply weighting by class frequency is suboptimal. Recent work suggests class difficulty is a better indicator for weighting, but existing methods use heuristic formulations for difficulty that don't generalize well across different datasets. The paper aims to address this by learning an optimal difficulty prediction strategy tailored to the specific dataset.

Methods

The core of the proposed method is Difficulty-Net, a meta-learning framework designed to predict class difficulty.

  • Difficulty-Net Architecture: It's a neural network (specifically, an MLP with two hidden layers) that takes the model's classification accuracies for all classes as input and outputs a difficulty score for each class. This design inherently considers relative difficulty, as the prediction for one class is influenced by the accuracies of all other classes.
  • Meta-Learning Objective: The system is trained using a meta-learning objective. This involves optimizing the parameters of Difficulty-Net ($\theta$) and the main classifier network ($\phi$) iteratively.
  • Driver Loss ($L_{dr}$): A crucial innovation is the "driver loss." The authors found that simply learning difficulty from accuracy wasn't enough; the model needed guidance. The driver loss encourages Difficulty-Net to assign higher difficulty scores to classes with lower accuracies, pushing the model towards a more meaningful learning direction.
  • Relative Difficulty: Unlike previous methods that used absolute difficulty (e.g., $1 - accuracy$), Difficulty-Net focuses on relative difficulty. This means a class's difficulty is assessed in context with other classes, allowing for more nuanced weighting. For instance, an "easy" class with high accuracy might still be assigned a high difficulty if other classes are even easier.
  • Learning Process: The training involves a three-step iterative process:
  • Update classifier parameters ($\phi$) using current Difficulty-Net weights.
  • Update Difficulty-Net parameters ($\theta$) using a meta-learning loss and the new driver loss.
  • Update classifier parameters ($\phi$) again using the updated Difficulty-Net weights.

Datasets

The paper evaluates its method on several standard long-tailed datasets:

  • CIFAR100-LT: A long-tailed version of CIFAR100, with imbalance ratios ranging from 10 to 200.
  • ImageNet-LT: A long-tailed version of ImageNet, with a heavy imbalance of 256.
  • Places-LT: A long-tailed subset of Places-2, with an imbalance of 996.

The authors also mention using a "meta-dataset" ($S_{meta}$) for meta-learning, constructed similarly to previous meta-learning methods. They note that this meta-dataset can also serve as a validation set ($S_{val}$), meaning no additional data is required.

Key Insights and Ideas

  • Relative Difficulty is Key: The paper strongly argues and empirically demonstrates that considering class difficulty relative to other classes is more effective than using absolute difficulty. This is because it allows for a more nuanced understanding of which classes truly require more attention in a long-tailed scenario.
  • Heuristic Difficulty Formulas are Insufficient: The authors show that fixed formulas for class difficulty (like $1-accuracy$) don't perform optimally across all datasets. This motivates the need for a learned approach.
  • The "Driver Loss" is Essential: Simply learning difficulty from accuracy and meta-learning wasn't enough. The introduction of the "driver loss" was critical for guiding Difficulty-Net to learn meaningful difficulty predictions, preventing it from assigning similar difficulties to all classes.
  • Class-Level Weighting is Superior: The paper reinforces the finding that class-level weighting is more effective than sample-level weighting for long-tailed recognition, especially in scenarios with high imbalance.

Key Results

  • State-of-the-Art Performance: Difficulty-Net achieved state-of-the-art results on multiple long-tailed datasets, including CIFAR100-LT and ImageNet-LT, particularly when combined with decoupled learning strategies.
  • CIFAR100-LT: In end-to-end training, Difficulty-Net achieved 47.96% accuracy with an imbalance of 100 and 63.52% with an imbalance of 10. When combined with decoupled learning methods like LAS, it reached 48.32% and 65.22% respectively, outperforming other methods.
  • ImageNet-LT: Difficulty-Net combined with LAS achieved 54.0% accuracy on ResNet-50 in decoupled learning, setting a new state-of-the-art.
  • Places-LT: Difficulty-Net combined with LAS achieved 41.7% overall accuracy, with significant gains in medium- and few-shot accuracies.
  • Ablation Studies:
    • Relative difficulty significantly outperformed absolute difficulty (e.g., 47.96% vs. 46.81% on CIFAR100-LT, imb=100).
  • The driver loss was crucial; without it, performance dropped significantly, especially in high imbalance scenarios.
  • Class-level difficulty prediction was superior to sample-level prediction.

Limitations and Future Work

  • Hyperparameter Tuning: The paper mentions that the hyperparameter $\lambda$ (controlling the driver loss) needs tuning, and higher values are better for higher imbalance, but too high a value can degrade performance. They found $\lambda=0.3$ to be consistently good.
  • Computational Cost: While not explicitly detailed as a limitation, meta-learning typically involves more complex training procedures than standard supervised learning. The authors mention calculating accuracy only after each epoch to save computational time.
  • Future Work: The paper suggests that their method can be easily combined with ensemble techniques (like Mixture of Experts), and they verified that it indeed provides significant performance gains when used in such settings. Further analysis of the learned difficulty distributions and their relationship to model performance is also implied.

Summary updated

BibTeX
@inproceedings{sinha2023difficulty,
  author = {Saptarshi Sinha and Hiroki Ohashi},
  title = {{Difficulty-Net: Learning To Predict Difficulty for Long-Tailed Recognition}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure