Introduction
In the field of computer vision, a persistent challenge is the "granularity gap." While large-scale datasets with coarse labels (e.g., "butterfly," "car") are relatively easy to compile, obtaining expert-level annotations for fine-grained sub-categories (e.g., specific butterfly species or car models) is expensive and difficult to scale. Standard supervised learning often fails in these scenarios; models tend to "collapse" all sub-categories into a single representation, making them ineffective for fine-grained retrieval or classification tasks.
The Grafit Framework
Grafit (GRAnularity-FIne-Tuning) is a novel training procedure that enables models to learn fine-grained features using only coarse labels. The method achieves this by preserving the underlying "latent" structure of the data through a dual-loss objective applied to a standard convolutional trunk, such as a ResNet-50.
Dual-Loss Objective
The core of Grafit is the combination of a supervised signal with a self-supervised regularizer:
- Supervised kNN Loss ($L_{knn}$): Unlike standard cross-entropy loss which uses a fixed softmax layer, Grafit employs a neighborhood-based loss. This encourages images with the same coarse label to cluster together in the embedding space, making the representation inherently suited for k-Nearest Neighbor (kNN) retrieval.
- Self-Supervised Instance Loss ($L_{inst}$): Inspired by methods like BYOL, this loss forces different augmentations of the same image to have similar embeddings. This acts as a crucial regularizer that prevents the coarse labels from pulling different sub-classes too close together, ensuring the model maintains "instance-level" distinctness.
Conditional Ranking
For retrieval tasks at test time, Grafit introduces a scoring function that combines cosine similarity with the model's own confidence in the coarse category. This "filters" the search space, ensuring that retrieved fine-grained results are semantically consistent with the predicted coarse category, further bridging the gap between coarse training and fine-grained inference.
Experimental Evaluation
The authors evaluated Grafit across several benchmarks with hierarchical labels, including CIFAR-100, ImageNet (using 127 coarse categories based on WordNet), and the iNaturalist biological datasets.
Key Results
- On-the-fly Classification: On ImageNet, Grafit improved top-1 accuracy by +16.3% over standard baselines when classifying 1,000 fine classes using only 127 coarse labels for training.
- State-of-the-Art Retrieval: On CIFAR-100, the mean Average Precision (mAP) for fine-grained retrieval increased from 42.5% (baseline) to 55.7%.
- Taxonomic Robustness: The method is remarkably robust across biological hierarchies. Training on "Genus" labels still allows the model to separate "Species," and even training on "Family" provides a significant boost over standard baselines.
- Transfer Learning: Using a RegNetY-8.0GF backbone, Grafit established new state-of-the-art results on iNaturalist 2018 (81.2%) and Oxford Flowers-102 (99.1%).
Discussion and Limitations
A key insight of this work is that self-supervised instance learning provides the "fine-grained" signal that coarse labels lack. While coarse labels provide global structure, the instance loss ensures local structure is preserved.
However, some limitations remain. kNN-based methods require storing a memory bank of embeddings for all training images, which can be a bottleneck for billion-scale datasets. Additionally, searching a kNN database is inherently slower than standard linear classification, though this can be mitigated with approximate nearest neighbor (ANN) libraries like Faiss. Future work may explore handling noisy hierarchies or discovering fine-grained relationships in a completely unsupervised manner.
Conclusion
Grafit presents a powerful approach to fine-grained representation learning. By optimizing for kNN performance and incorporating self-supervised instance signals, it allows models to handle "on-the-fly" classification—where new fine-grained labels are introduced at test time—without needing to retrain the final layer.
Summary updated