Grafit: Learning fine-grained image representations with coarse labels

Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, Hervé Jégou

Introduction

In the field of computer vision, a persistent challenge is the "granularity gap." While large-scale datasets with coarse labels (e.g., "butterfly," "car") are relatively easy to compile, obtaining expert-level annotations for fine-grained sub-categories (e.g., specific butterfly species or car models) is expensive and difficult to scale. Standard supervised learning often fails in these scenarios; models tend to "collapse" all sub-categories into a single representation, making them ineffective for fine-grained retrieval or classification tasks.

Grafit addresses the granularity gap by learning to distinguish fine-grained species even when trained on coarse genus labels.

The Grafit Framework

Grafit (GRAnularity-FIne-Tuning) is a novel training procedure that enables models to learn fine-grained features using only coarse labels. The method achieves this by preserving the underlying "latent" structure of the data through a dual-loss objective applied to a standard convolutional trunk, such as a ResNet-50.

Dual-Loss Objective

The core of Grafit is the combination of a supervised signal with a self-supervised regularizer:

  1. Supervised kNN Loss ($L_{knn}$): Unlike standard cross-entropy loss which uses a fixed softmax layer, Grafit employs a neighborhood-based loss. This encourages images with the same coarse label to cluster together in the embedding space, making the representation inherently suited for k-Nearest Neighbor (kNN) retrieval.
  2. Self-Supervised Instance Loss ($L_{inst}$): Inspired by methods like BYOL, this loss forces different augmentations of the same image to have similar embeddings. This acts as a crucial regularizer that prevents the coarse labels from pulling different sub-classes too close together, ensuring the model maintains "instance-level" distinctness.
The Grafit architecture utilizes two paths to optimize for both instance recognition and kNN classifier performance.

Conditional Ranking

For retrieval tasks at test time, Grafit introduces a scoring function that combines cosine similarity with the model's own confidence in the coarse category. This "filters" the search space, ensuring that retrieved fine-grained results are semantically consistent with the predicted coarse category, further bridging the gap between coarse training and fine-grained inference.

Experimental Evaluation

The authors evaluated Grafit across several benchmarks with hierarchical labels, including CIFAR-100, ImageNet (using 127 coarse categories based on WordNet), and the iNaturalist biological datasets.

Key Results

  • On-the-fly Classification: On ImageNet, Grafit improved top-1 accuracy by +16.3% over standard baselines when classifying 1,000 fine classes using only 127 coarse labels for training.
  • State-of-the-Art Retrieval: On CIFAR-100, the mean Average Precision (mAP) for fine-grained retrieval increased from 42.5% (baseline) to 55.7%.
  • Taxonomic Robustness: The method is remarkably robust across biological hierarchies. Training on "Genus" labels still allows the model to separate "Species," and even training on "Family" provides a significant boost over standard baselines.
  • Transfer Learning: Using a RegNetY-8.0GF backbone, Grafit established new state-of-the-art results on iNaturalist 2018 (81.2%) and Oxford Flowers-102 (99.1%).

Discussion and Limitations

A key insight of this work is that self-supervised instance learning provides the "fine-grained" signal that coarse labels lack. While coarse labels provide global structure, the instance loss ensures local structure is preserved.

However, some limitations remain. kNN-based methods require storing a memory bank of embeddings for all training images, which can be a bottleneck for billion-scale datasets. Additionally, searching a kNN database is inherently slower than standard linear classification, though this can be mitigated with approximate nearest neighbor (ANN) libraries like Faiss. Future work may explore handling noisy hierarchies or discovering fine-grained relationships in a completely unsupervised manner.

Conclusion

Grafit presents a powerful approach to fine-grained representation learning. By optimizing for kNN performance and incorporating self-supervised instance signals, it allows models to handle "on-the-fly" classification—where new fine-grained labels are introduced at test time—without needing to retrain the final layer.

Summary updated

Abstract

This paper tackles the problem of learning a finer representation than the one provided by training labels. This enables fine-grained category retrieval of images in a collection annotated with coarse labels only.Our network is learned with a nearest-neighbor classifier objective, and an instance loss inspired by self-supervised learning. By jointly leveraging the coarse labels and the underlying fine-grained latent space, it significantly improves the accuracy of category-level retrieval methods.Our strategy outperforms all competing methods for retrieving or classifying images at a finer granularity than that available at train time. It also improves the accuracy for transfer learning tasks to fine-grained datasets.
BibTeX
@misc{touvron2020grafit,
  author = {Hugo Touvron and Alexandre Sablayrolles and Matthijs Douze and Matthieu Cord and Hervé Jégou},
  title = {{Grafit: Learning fine-grained image representations with coarse labels}},
  year = {2020},
  eprint = {2011.12982},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.12982},
}

Loading related content...

Figure