Enhancing Visual Embeddings through Weakly Supervised Captioning for Zero-Shot Learning

Matteo Bustreo, Jacopo Cavazza, Vittorio MurinoWorkshops @ ICCV 2019 · OpenReview

Abstract

Visual features designed for image classification have shown to be useful in zero-shot learning (ZSL) when generalizing towards classes not seen during training. In this paper, we argue that a more effective way of building visual features for ZSL is to extract them through captioning, in order not just to classify an image but, instead, to describe it. However, modern captioning models rely on a massive level of supervision, e.g up to 15 extended descriptions per instance provided by humans, which is simply not available for ZSL benchmarks. In the latter in fact, the available annotations inform about the presence/absence of attributes within a fixed list only. Worse, attributes are seldom annotated at the image level, but rather, at the class level only: because of this, the annotation cannot be visually grounded. In this paper, we deal with such a weakly supervised regime to train an end-to-end LSTM captioner, whose backbone CNN image encoder can provide better features for ZSL. Our enhancement of visual features, called "VisEn", is compatible with any generic ZSL method, without requiring changes in its pipeline. Experimentally, VisEn is capable of sharply improving recognition performance on unseen classes, as we demonstrate through an ablation study which encompasses different ZSL approaches. Further, on the challenging fine-grained CUB dataset, VisEn improves by margin state-of-the-art methods, by using visual descriptors of one order of magnitude smaller.

Introduction

Zero-Shot Learning (ZSL) presents a unique challenge: recognizing objects from categories that were never encountered during the training phase. To bridge the gap between seen and unseen classes, ZSL models rely on visual embeddings—mathematical representations of images—that are mapped to a shared semantic space defined by attributes (e.g., "has stripes," "is furry").

Traditionally, these visual embeddings are extracted from CNNs trained for large-scale image classification, such as ResNet on ImageNet. However, classification-based features are inherently designed to discard "irrelevant" details to arrive at a single label. In ZSL, these discarded details—such as the specific curvature of a beak or the texture of a wing—are precisely the discriminative features needed to identify a new species. This paper proposes VisEn (Visual Enhancement), a method that shifts the training objective from classification to description.

The VisEn Framework

The core idea of VisEn is to train a CNN encoder as part of a captioning model rather than a classification model. By forcing the encoder to provide enough information for a decoder to describe an image's attributes, the resulting embeddings become more semantically dense.

The VisEn architecture: A ResNet-152 encoder produces embeddings that are decoded by an LSTM to predict a sequence of attributes.

Architecture and Training

The framework utilizes a standard CNN+LSTM architecture. A ResNet-152 acts as the encoder, and a uni-directional LSTM serves as the decoder. Instead of using the final classification layer, VisEn extracts the 256-dimensional (or 2048-dimensional) vector that serves as the initial state for the LSTM. This vector is pressured to encapsulate all semantic information necessary for the LSTM to generate a list of attributes.

Because standard ZSL datasets lack the natural language sentences typical of captioning tasks, the authors employ a weakly supervised regime. For datasets like CUB, they use instance-level attribute annotations. For datasets like AWA2, where attributes are only known at the class level, they introduce a "negative supervision" trick: using semantic incompatibility to inform the model about what is not present in an image (e.g., confirming a zebra image lacks "wings").

Experimental Results and Key Insights

The authors evaluated VisEn on two primary benchmarks: the fine-grained Caltech-UCSD Birds 200 (CUB) and the coarse-grained Animals with Attributes 2 (AWA2).

Performance Boosts

VisEn features were integrated into several popular ZSL algorithms, including SynC, SAE, and CDL. In nearly every instance, the new embeddings led to a sharp increase in performance without requiring modifications to the underlying algorithms. On the CUB dataset, VisEn established a new state-of-the-art for inductive ZSL, outperforming previous methods by +9.7% on standard splits.

Semantic Density and Visual Grounding

A significant finding was that 256-dimensional VisEn features often outperformed standard 2048-dimensional features. This suggests that the quality and semantic relevance of the embedding are more critical than its dimensionality.

t-SNE visualization comparing default ResNet features (bottom) with VisEn enhanced embeddings (top), showing better semantic clustering.

Qualitative analysis via t-SNE visualizations (as seen above) demonstrates that VisEn features cluster images based on actual visual traits, such as wing color, rather than background noise or species labels alone. This improved visual grounding is key to the model's superior generalization to unseen classes.

Conclusion

VisEn demonstrates that training visual encoders to "describe" rather than just "label" produces embeddings that are significantly more effective for Zero-Shot Learning. While the method benefits most from instance-level supervision, even class-level weak supervision provides a meaningful performance boost. Future work may explore more complex decoder architectures or the integration of natural language descriptions to further refine these semantically rich visual representations.

Summary updated

Abstract

Visual features designed for image classification have shown to be useful in zero-shot learning (ZSL) when generalizing towards classes not seen during training. In this paper, we argue that a more effective way of building visual features for ZSL is to extract them through captioning, in order not just to classify an image but, instead, to describe it. However, modern captioning models rely on a massive level of supervision, e.g up to 15 extended descriptions per instance provided by humans, which is simply not available for ZSL benchmarks. In the latter in fact, the available annotations inform about the presence/absence of attributes within a fixed list only. Worse, attributes are seldom annotated at the image level, but rather, at the class level only: because of this, the annotation cannot be visually grounded. In this paper, we deal with such a weakly supervised regime to train an end-to-end LSTM captioner, whose backbone CNN image encoder can provide better features for ZSL. Our enhancement of visual features, called "VisEn", is compatible with any generic ZSL method, without requiring changes in its pipeline (a part from adapting hyper-parameters). Experimentally, VisEn is capable of sharply improving recognition performance on unseen classes, as we demonstrate thorough an ablation study which encompasses different ZSL approaches. Further, on the challenging fine-grained CUB dataset, VisEn improves by margin state-of-the-art methods, by using visual descriptors of one order of magnitude smaller.
BibTeX
@inproceedings{bustreo2019enhancing,
  author = {Matteo Bustreo and Jacopo Cavazza and Vittorio Murino},
  title = {{Enhancing Visual Embeddings through Weakly Supervised Captioning for Zero-Shot Learning}},
  booktitle = {Workshops @ ICCV},
  year = {2019},
  url = {https://openreview.net/forum?id=fNM9X8gVgM9},
}
Figure