Abstract: Zero-shot classification is a generalization task where no instance from the target classes is seen during training. To allow for test-time transfer, each class is annotated with semantic information, commonly in the form of attributes or text descriptions. While classical zero-shot learning does not explicitly forbid using information from other datasets, the approaches that achieve the best absolute performance on image benchmarks rely on features extracted from encoders pretrained on Imagenet. This approach relies on hyper-optimized Imagenet-relevant parameters from the supervised classification setting, entangling important questions about the suitability of those parameters and how they were learned with more fundamental questions about representation learning and generalization. To remove these distractors, we propose a more challenging setting: Zero-Shot Learning from scratch (ZFS), which explicitly forbids the use of encoders fine-tuned on other datasets. Our analysis on this setting highlights the importance of local information, and compositional representations.
Introduction
Zero-Shot Learning (ZSL) aims to identify objects the model has never seen before by leveraging semantic descriptions, such as attributes. However, modern ZSL research often relies on features extracted from massive models pretrained on ImageNet. This "pretraining problem" creates entanglement, where it is unclear if a model's success is due to its reasoning capabilities or simply high-quality ImageNet features. Furthermore, it risks data leakage if test classes appeared in the pretraining set.
To address these issues, we introduce Zero-Shot Learning from scratch (ZFS). This stricter protocol requires models to learn representations—from low-level edges to high-level concepts—using only the training split of the target dataset, making it particularly relevant for specialized domains like medical or satellite imaging where large-scale pretraining data is unavailable.
Methodology: Local Compositional Representations
The core of our approach to ZFS is compositionality: the idea that an object is a collection of local parts (e.g., a bird defined by its "yellow beak" and "black wings"). We evaluate several representation learning frameworks, including Variational Auto-encoders (VAE), Adversarial Auto-encoders (AAE), and Deep InfoMax (DIM).
As shown in Figure 1, we introduce auxiliary local losses applied to the early layers of the encoder to force the extraction of semantically meaningful local features.
We implement two types of auxiliary classifiers:
- Local Label-based Classifier (LC): Predicts class labels based on local image patches.
- Local Attribute-based Classifier (AC): Maps local patches to semantic attributes using a prototypical network.
By adding these local losses ($\mathcal{L}_{local}$) to the standard model loss ($\mathcal{L}_{model}$), we encourage the model to look beyond global "blobs" and focus on reusable local components.
Experiments and Results
We evaluated our approach on three standard ZSL benchmarks—CUB (birds), AwA2 (animals), and SUN (scenes)—using the rigorous "Proposed Split" to ensure no overlap between training and testing concepts.
As detailed in Figure 2, adding local objectives almost universally improves performance in the ZFS setting:
- DIM Dominance: Deep InfoMax (DIM), which naturally values local structure by maximizing mutual information between patches and global features, performed exceptionally well. DIM with a local attribute classifier (AC) achieved the best results on AwA2 (43.62%) and SUN (35.35%).
- Locality is Key: In the absence of ImageNet pretraining, global features are often too noisy. Local features prove more "reusable" across different species, which is essential for zero-shot generalization.
- Competitive with Supervision: On the CUB dataset, the best ZFS models achieved accuracies (around 32-34%) comparable to a fully supervised label classifier, proving that learning from scratch is viable with the right inductive biases.
Conclusion
Our analysis of the ZFS setting demonstrates that encouraging local, compositional representations is the key to recognizing unseen objects without the ImageNet crutch. While challenges remain in scaling ZFS to deeper architectures and balancing global vs. local information, this protocol provides a much clearer picture of robust representation learning. Future work will explore more sophisticated ways to define "local parts" beyond fixed grid patches.
Summary updated