Abstract
Training robust image classifiers is notoriously difficult when labeled image data is scarce. This paper introduces a joint Intermodal and Intramodal Label Transfer (I2LT) method that leverages abundant, semantically rich text data to help annotate images, while using available labeled images as a "watchdog" to prevent irrelevant text transfers. The approach excels in extreme low-data regimes and seamlessly extends to zero-shot learning, allowing the model to classify images of entirely unseen categories using only text descriptions.
Introduction
In machine learning, text classification is generally easier than image classification because text features (words) map directly to semantic concepts, whereas visual features (pixels, edges) suffer from a massive "semantic gap." While we have an abundance of labeled text data, labeled image data is often scarce and expensive to obtain.
Previous research has attempted intermodal label transfer—using a learned bridge between text and images to pass labels from a rich text corpus to an unlabeled image. However, this approach is prone to negative transfer: if a test image's visual appearance doesn't align well with the source text descriptions, blindly transferring the text label will result in a misclassification. The authors tackle this by asking: How can we safely transfer knowledge from text to images without falling victim to negative transfer when the modalities don't perfectly align?
Methods
The authors propose I2LT, a framework that jointly optimizes two types of label transfer to classify a test image $\mathbf{z}$:
- Intermodal Transfer (Text $\rightarrow$ Image): The model learns a transfer function $T(\mathbf{x}_i, \mathbf{z})$ that maps both text features $\mathbf{x}_i$ and image features $\mathbf{z}$ into a shared "latent topic space." This bridge is built using co-occurrence data. To prevent overfitting, a trace norm regularizer is applied to ensure parsimony.
- Intramodal Transfer (Image $\rightarrow$ Image): To prevent negative transfer, the model incorporates a kernel-based transfer $K(\mathbf{z}_j, \mathbf{z})$ from a small auxiliary set of labeled training images. This acts as a "watchdog"—if the text description fails to capture the visual reality, the image-based labels take over.
The final prediction $f(\mathbf{z})$ is a combination of both transfers: $f(\mathbf{z}) = f_{\text{inter}}(\mathbf{z}) + f_{\text{intra}}(\mathbf{z})$. For zero-shot learning, the intramodal component is omitted.
Experimental Evaluation
The authors evaluated I2LT on several datasets, including Flickr-Wiki, NUS-WIDE, and fine-grained datasets like CUB200 Birds and Oxford Flower-102.
Low-Data Performance
On the Flickr-Wiki dataset, I2LT was tested in extreme low-data regimes (1 to 10 training images per category). It significantly outperformed baselines like standard SVMs, Heterogeneous Transfer Learning (HTL), and pure text-to-image transfer (TTI).
Zero-Shot Learning
On CUB200 and Flower-102, I2LT achieved average AUCs of 0.67 and 0.73, respectively, outperforming state-of-the-art zero-shot baselines. On the Extended CUB dataset, I2LT achieved 55.3% accuracy, significantly higher than standard embedding approaches like Word2Vec (28.4%) and GloVe (24.2%).
Discussion and Conclusion
The "watchdog" effect of combining modalities successfully mitigates negative transfer. Furthermore, minimizing the rank of the transfer matrix via the trace norm forces the model to find a concise semantic translation that generalizes well. While the current objective function is non-convex and inference is slightly more computationally expensive than simple baselines, the I2LT framework provides a robust solution for bridging the semantic gap. Future work may extend this modality-agnostic framework to audio-video or cross-lingual tasks.
Summary updated