Joint Intermodal and Intramodal Label Transfers for Extremely Rare or Unseen Classes

Guo-Jun Qi, Wei Liu, Charu C. Aggarwal, Thomas S. Huang

TL;DR

Training robust image classifiers is notoriously difficult when labeled image data is scarce. This paper introduces a joint Intermodal and Intramodal Label Transfer (I2LT) method that leverages abundant, semantically rich text data to help annotate images, while using available labeled images as a "watchdog" to prevent irrelevant text transfers. The approach excels in extreme low-data regimes and seamlessly extends to zero-shot learning, allowing the model to classify images of entirely unseen categories using only text descriptions.

What Problem Does This Paper Address?

In machine learning, text classification is generally easier than image classification because text features (words) map directly to semantic concepts, whereas visual features (pixels, edges) suffer from a massive "semantic gap." While we have an abundance of labeled text data, labeled image data is often scarce and expensive to obtain.

Previous research has attempted intermodal label transfer—using a learned bridge between text and images to pass labels from a rich text corpus to an unlabeled image. However, this approach is prone to negative transfer: if a test image's visual appearance doesn't align well with the source text descriptions, blindly transferring the text label will result in a misclassification. The authors tackle this by asking: How can we safely transfer knowledge from text to images without falling victim to negative transfer when the modalities don't perfectly align?

Methods

To solve this, the authors propose I2LT, a framework that jointly optimizes two types of label transfer:

  1. Intermodal Transfer (Text $\rightarrow$ Image): The model learns a transfer function that maps both text and image features into a shared, hypothetical "latent topic space." This bridge is built using readily available co-occurrence data (e.g., images and their surrounding text on a webpage). To prevent overfitting on noisy web data, the authors apply the principle of parsimony using a trace norm regularizer, which forces the model to explain the text-image alignment using the fewest possible latent topics.
  2. Intramodal Transfer (Image $\rightarrow$ Image): To prevent negative transfer, the model also incorporates a traditional kernel-based transfer from a small auxiliary set of labeled training images. This acts as a "watchdog"—if the text description fails to capture the visual reality of the test image, the image-based labels take over.
  3. Joint Optimization: The final prediction is a combination of both transfers. Because the combined objective function (using a hyperbolic tangent activation and trace norm) is non-convex, the authors optimize it using an alternating proximal gradient method.
  4. Zero-Shot Extension: For classes with zero training images, the model drops the intramodal transfer and relies entirely on the learned intermodal bridge to pass labels from text descriptions (like Wikipedia articles) to unseen test images.

Datasets

The authors evaluated their approach on several datasets, ranging from general web images to fine-grained classification tasks:

  • Flickr-Wiki Dataset: A custom dataset of 10 categories constructed by crawling images and co-occurring text from Flickr and Wikipedia.
  • NUS-WIDE: A real-world dataset of 269,648 Flickr images with tags, evaluated across 81 concepts using CNN features.
  • CUB200 Birds & Oxford Flower-102: Standard fine-grained datasets (200 bird species and 102 flower species) paired with Wikipedia/BBC text articles, used specifically to test the zero-shot learning capabilities.
  • Extended CUB: A larger variant of the CUB dataset (11,788 images) used to compare against modern output embedding models.

Key Insights and Ideas

  • The "Watchdog" Effect is Real: Relying solely on cross-modal (text-to-image) transfer is risky. By jointly optimizing text-to-image and image-to-image transfers, the model successfully mitigates negative transfer, yielding higher accuracy than using either modality alone.
  • Parsimony is Crucial for Semantic Bridges: When bridging heterogeneous feature spaces (like sparse text vectors and dense visual features), simpler is better. Minimizing the rank of the transfer matrix (via the trace norm) forces the model to find a concise semantic translation, which generalizes much better to unseen data.
  • Shared Transfer Matrices for Zero-Shot: In the zero-shot scenario, the authors found that learning a class-independent transfer matrix—shared across both seen and unseen classes—allows the model to fully exploit the image labels of seen classes to build a better text-image bridge for the unseen ones.

Key Results

  • Extreme Low-Data Regime: On the Flickr-Wiki dataset, when restricted to just 2 or 10 training images per category, I2LT significantly outperformed baselines like standard SVMs, Heterogeneous Transfer Learning (HTL), and pure text-to-image transfer (TTI).
  • NUS-WIDE: I2LT achieved a Mean Average Precision (MAP) of 0.4042, nearly doubling the performance of a baseline SVM (0.2078) and beating the pure intermodal transfer model (0.3418). It won on 67 out of 81 concepts.
  • Zero-Shot Learning: On CUB200 and Flower-102, I2LT achieved average AUCs of 0.67 and 0.73, respectively, outperforming state-of-the-art zero-shot baselines like Twin Gaussian Processes and Write A Classifier (WAC).
  • Embedding Comparison: On the Extended CUB dataset, I2LT achieved 55.3% accuracy, crushing standard embedding approaches like Word2Vec (28.4%) and GloVe (24.2%) by leveraging multiple source labels rather than just pairwise compatibility.

Limitations and Future Work

  • Non-Convexity: The objective function for the joint transfer is non-convex. While the proximal gradient method reliably decreases the objective in each iteration, it is only guaranteed to converge to a stationary point, which may not be the global optimum.
  • Computational Cost: Because I2LT calculates both intermodal and intramodal transfers during inference, its testing time is slightly higher than simpler baselines (e.g., 29.3 seconds vs. 8.2 seconds for a standard SVM on the same hardware).
  • Future Directions: While the paper focuses on text-to-image transfer, the authors note the mathematical framework is modality-agnostic. Future work could apply this to audio-video recognition or cross-lingual document classification.

Summary updated

Abstract

In this paper, we present a label transfer model from texts to images for image classification tasks. The problem of image classification is often much more challenging than text classification. On one hand, labeled text data is more widely available than the labeled images for classification tasks. On the other hand, text data tends to have natural semantic interpretability, and they are often more directly related to class labels. On the contrary, the image features are not directly related to concepts inherent in class labels. One of our goals in this paper is to develop a model for revealing the functional relationships between text and image features as to directly transfer intermodal and intramodal labels to annotate the images. This is implemented by learning a transfer function as a bridge to propagate the labels between two multimodal spaces. However, the intermodal label transfers could be undermined by blindly transferring the labels of noisy texts to annotate images. To mitigate this problem, we present an intramodal label transfer process, which complements the intermodal label transfer by transferring the image labels instead when relevant text is absent from the source corpus. In addition, we generalize the inter-modal label transfer to zero-shot learning scenario where there are only text examples available to label unseen classes of images without any positive image examples. We evaluate our algorithm on an image classification task and show the effectiveness with respect to the other compared algorithms.
BibTeX
@article{qi2017joint,
  author = {Guo-Jun Qi and Wei Liu and Charu C. Aggarwal and Thomas S. Huang},
  title = {{Joint Intermodal and Intramodal Label Transfers for Extremely Rare or Unseen Classes}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2017},
  url = {https://openreview.net/forum?id=SIQ4esS_Pe9},
}
Figure