Enhancing Instance-Level Image Classification with Set-Level Labels

Renyu Zhang, Aly A Khan, Yuxin Chen, Robert L. Grossman

In the world of machine learning, high-quality labels are invaluable but notoriously expensive to acquire. Traditional supervised learning and Few-Shot Learning (FSL) typically require meticulously labeled individual instances to train effective models. However, in many real-world scenarios, we have an abundance of "coarse" labels at the group or set level, but very few "fine-grained" labels for individual items.

Consider a digital photo library: it is easy to label an entire album as "Picnic at the Park" (set-level), but labeling every single blade of grass, sandwich, or person in every photo (instance-level) is a nightmare. Similarly, in medicine, a pathologist might label an entire tissue slide by the organ it came from (coarse), but labeling every specific cell type within that slide (fine-grained) is incredibly time-consuming.

Examples of coarse-grained set labels versus fine-grained instance labels in natural and medical images.

To tackle the challenge of using "cheap" coarse labels to help models perform better on "expensive" fine-grained classification tasks, researchers have introduced FACILE (Fine-grAined representation learning from Coarse-graIned LabEls). This novel framework uses easily obtainable set-level labels to boost the performance of instance-level classification, achieving massive accuracy improvements over state-of-the-art baselines.

The FACILE Framework

FACILE bridges the gap between Multiple Instance Learning (MIL) and standard classification by treating pre-training as a set-based problem to extract instance-level features. It operates in two main stages:

The FACILE framework: Pre-training with coarse-grained labels and fine-tuning with fine-grained labels.
  1. Pre-training (The Coarse Phase): The model is trained on a source domain using only set-level labels. It uses a feature map to extract characteristics from individual instances and a "set-input model" to aggregate these into a set-level representation. The authors tested this using both Supervised Contrastive Learning (SupCon) and Fully Supervised Pre-training (FSP).
  2. Fine-tuning (The Fine Phase): The pre-trained feature map is then frozen or adapted to a target domain where only a few fine-grained labels are available (the FSL setting).

Unlike standard transfer learning that requires fine-grained labels in the source domain, FACILE proves that you can learn high-quality, discriminative features using only coarse, set-level supervision. The authors also provide a rigorous theoretical proof showing that this approach achieves a "fast" excess risk convergence rate of $O(1/n)$, meaning the model learns very efficiently from limited fine-grained data.

Datasets and Key Results

The researchers validated FACILE across two very different domains. For natural images, they used CIFAR-100, grouping its 100 classes into 20 superclasses to simulate coarse labels. For histopathology (medical imaging), they used TCGA and GTEx datasets for coarse source labels (organ/tumor types), and LC25000, PAIP, NCT-CRC-HE-100K, and a private Pancreatic (PDAC) dataset for fine-grained target labels.

Key Insights and Breakthroughs: * Medical Breakthrough: On the LC25000 histopathology benchmark, FACILE achieved a 13% improvement in classification accuracy compared to the strongest existing baseline. * Superior to Self-Supervision: FACILE consistently outperformed popular self-supervised methods like SimCLR and SimSiam. This suggests that even "coarse" supervision is better than no supervision at all for learning useful representations. * Hierarchy is a Feature: Coarse labels aren't just "bad" labels; they capture essential structural relationships. An organ-level label provides the context that makes identifying specific tissue types easier. * Foundation Model Compatibility: FACILE can be used to "supercharge" existing foundation models like CLIP and DINO V2, making them significantly more accurate for specialized tasks like anomaly detection or medical diagnosis.

Limitations and Future Work

While FACILE is powerful, the authors note a few hurdles. Pre-training on massive sets of images (like whole-slide pathology images) is computationally expensive and requires significant memory. Furthermore, the current version uses all available coarse data. Future research should focus on "active" or "selective" pre-training—identifying which coarse sets are most informative to reduce training time. Finally, while they tested several architectures (Deep Sets, Set Transformers), no single architecture was a "silver bullet" for every dataset, suggesting that the choice of set-aggregator is still task-dependent.

Summary updated

Abstract

Instance-level image classification tasks have traditionally relied on single-instance labels to train models, e.g., few-shot learning and transfer learning. However, set-level coarse-grained labels that capture relationships among instances can provide richer information in real-world scenarios. In this paper, we present a novel approach to enhance instance-level image classification by leveraging set-level labels. We provide a theoretical analysis of the proposed method, including recognition conditions for fast excess risk rate, shedding light on the theoretical foundations of our approach. We conducted experiments on two distinct categories of datasets: natural image datasets and histopathology image datasets. Our experimental results demonstrate the effectiveness of our approach, showcasing improved classification performance compared to traditional single-instance label-based methods. Notably, our algorithm achieves 13\% improvement in classification accuracy compared to the strongest baseline on the histopathology image classification benchmarks. Importantly, our experimental findings align with the theoretical analysis, reinforcing the robustness and reliability of our proposed method. This work bridges the gap between instance-level and set-level image classification, offering a promising avenue for advancing the capabilities of image classification models with set-level coarse-grained labels.
BibTeX
@inproceedings{zhang2024enhancing,
  author = {Renyu Zhang and Aly A Khan and Yuxin Chen and Robert L. Grossman},
  title = {{Enhancing Instance-Level Image Classification with Set-Level Labels}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=AZW3qlCGTe},
}
Figure