Coarse-to-Fine Curriculum Learning

Otilia Stretcu, Emmanouil Antonios Platanios, Tom M. Mitchell, Barnabás PóczosarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel "coarse-to-fine" curriculum learning approach for classification tasks. Instead of training on the final, complex task directly, it breaks it down into a sequence of progressively finer sub-tasks, leveraging a learned hierarchy of class similarities. This method significantly boosts performance, especially for datasets with many classes, by enabling models to learn more effectively through intermediate, simpler goals.

What Problem Does This Paper Address?

Machine learning models are typically trained to solve a target task directly, using all available data. This contrasts with human learning, where we often learn complex skills by first mastering simpler, intermediate steps. For instance, a child learns to distinguish animals from objects before learning to differentiate between specific breeds of dogs. This paper addresses the gap in machine learning by proposing a method that mimics this human-like, hierarchical learning process. Specifically, it tackles the challenge of training models for classification tasks with a large number of classes, where direct training can be inefficient and lead to confusion between similar classes. The authors argue that existing curriculum learning methods primarily focus on the input data distribution (e.g., presenting easier examples first), while their approach focuses on structuring the output space of the model.

Methods

The core of the proposed method is a "coarse-to-fine" curriculum learning strategy that operates on the output space.

  1. Automatic Hierarchy Generation: The algorithm first constructs a hierarchy of class similarities. This is done by training a baseline model and then using its class embedding similarity (derived from the final layer's weights) to guide a hierarchical clustering algorithm. This ensures that classes that the model finds confusing are grouped together at coarser levels.
  2. Coarse-to-Fine Task Sequence: The generated hierarchy defines a sequence of classification tasks. The coarsest task involves classifying into broad categories (e.g., "animal" vs. "object"), while subsequent tasks progressively refine these categories until the original, fine-grained classes are reached.
  3. Knowledge Transfer: The model is trained sequentially on these tasks. Crucially, the parameters learned for a coarser task are used to initialize the model for the next, finer task. This allows the model to transfer acquired knowledge, effectively "pre-training" on easier sub-problems before tackling the full complexity. The paper proposes a "continuous" variant where a single model's parameters are updated across all levels, and a "staged" variant where separate models are trained at each level, with knowledge transferred via parameter initialization.

Datasets

The authors evaluated their method on:

  • Synthetic Dataset (Shapes): A custom dataset of 64x64 images featuring 10 geometric shapes in 3 colors, resulting in 30 classes. This dataset was designed to study the method's properties and compare different class similarity metrics.
  • Real Datasets:
    • CIFAR-10
    • CIFAR-100 (and a "Coarse" version with 20 predefined coarse classes)
    • Tiny-ImageNet
    • Subsets of ImageNet

Key Insights and Ideas

  • Output Space Curriculum is Powerful: The paper demonstrates that structuring the learning process based on the output space (class hierarchy) can be more effective than solely manipulating the input data distribution, especially for complex classification tasks.
  • Automatic Hierarchy Generation is Key: The ability to automatically generate a relevant class hierarchy based on model confusion (via class embedding similarity) is crucial. This avoids the need for manual annotation of class relationships and adapts to what the model actually finds difficult.
  • Coarse-to-Fine Learning Mimics Human Learning: The approach effectively breaks down complex problems into manageable steps, leading to better generalization and performance, mirroring how humans learn.
  • Transfer Learning is Essential: The sequential training and parameter transfer between hierarchy levels are critical for the method's success, allowing the model to build upon previously acquired knowledge.

Key Results

  • Significant Performance Gains: The proposed coarse-to-fine curriculum learning approach consistently outperformed baseline models across various datasets and neural network architectures (CNN, ResNet, WideResnet).
  • Impact on Many-Class Problems: The gains were particularly pronounced on datasets with a large number of classes (e.g., CIFAR-100, Tiny-ImageNet).
  • Effective in Low-Data Regimes: The method showed significant benefits when training data was scarce, as pre-training on coarser tasks provided a strong starting point.
  • Robustness to Curriculum Length: While there's an optimal curriculum length, the method generally showed robustness across a range of durations.
  • Comparison to Other Methods: The proposed approach outperformed other curriculum learning and hierarchical classification methods on several benchmarks.

Limitations and Future Work

  • Computational Cost: While the authors claim the computational complexity is manageable, the staged approach can be more computationally intensive than the continuous one. The initial training of a baseline model to derive class similarities also adds overhead, though they suggest this can be mitigated.
  • Hyperparameter Sensitivity (Continuous Variant): The continuous variant introduces a hyperparameter for the total curriculum length, which needs to be set.
  • Future Work: The authors suggest that their approach could be extended to other domains beyond classification. They also note that their method could be adapted for hierarchical classification by creating ensembles of classifiers trained at each level, though this is beyond the scope of the current paper. The paper also highlights the potential for misuse of any training strategy, a general ethical concern for AI.

Summary updated

Abstract

When faced with learning challenging new tasks, humans often follow sequences of steps that allow them to incrementally build up the necessary skills for performing these new tasks. However, in machine learning, models are most often trained to solve the target tasks directly.Inspired by human learning, we propose a novel curriculum learning approach which decomposes challenging tasks into sequences of easier intermediate goals that are used to pre-train a model before tackling the target task. We focus on classification tasks, and design the intermediate tasks using an automatically constructed label hierarchy. We train the model at each level of the hierarchy, from coarse labels to fine labels, transferring acquired knowledge across these levels. For instance, the model will first learn to distinguish animals from objects, and then use this acquired knowledge when learning to classify among more fine-grained classes such as cat, dog, car, and truck. Most existing curriculum learning algorithms for supervised learning consist of scheduling the order in which the training examples are presented to the model. In contrast, our approach focuses on the output space of the model. We evaluate our method on several established datasets and show significant performance gains especially on classification problems with many labels. We also evaluate on a new synthetic dataset which allows us to study multiple aspects of our method.
BibTeX
@misc{stretcu2021coarse,
  author = {Otilia Stretcu and Emmanouil Antonios Platanios and Tom M. Mitchell and Barnabás Póczos},
  title = {{Coarse-to-Fine Curriculum Learning}},
  year = {2021},
  eprint = {2106.04072},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.04072},
}
Figure