Full-Network Embedding in a Multimodal Embedding Pipeline

Armand Vilalta, Dario Garcia-Gasulla, Ferran Parés, Jonathan Moreno, Eduard Ayguadé, Jesús Labarta, Ulises Cortés, Toyotaro Suzumura

Abstract

This paper explores upgrading multimodal embedding pipelines by replacing standard single-layer CNN image features with a "Full-Network Embedding" (FNE). By capturing and discretizing multi-scale features from all layers of a neural network, the FNE significantly improves performance on image annotation and retrieval tasks across standard benchmarks.

Introduction

In vision-language tasks like image annotation (finding a fitting caption for an image) and image retrieval (finding an image for a text query), models typically map both modalities into a shared embedding space so their similarities can be measured.

To represent the image in this space, most approaches simply extract features from the final fully-connected layer of a pre-trained Convolutional Neural Network (CNN). However, this single-layer approach discards valuable lower-level and mid-level visual features learned earlier in the network. The authors investigate whether using a richer, multi-scale representation—specifically the Full-Network Embedding (FNE)—can improve the alignment between images and text in this shared multimodal space.

Methodology

The authors build on a well-known multimodal pipeline by Kiros et al., which uses Gated Recurrent Units (GRUs) to encode text and a pairwise ranking loss to pull matching image-text pairs closer together. The core novelty of this work is the replacement of the standard last-layer CNN features with the Full-Network Embedding (FNE).

The Full-Network Embedding pipeline architecture, illustrating the process from multi-layer feature extraction to discretization and multimodal mapping.

The FNE generation process works as follows:

  1. Multi-layer Extraction: It extracts activations from all convolutional and fully-connected layers of a pre-trained CNN.
  2. Spatial Pooling: To manage the massive amount of data from convolutional layers, it applies spatial average pooling to each filter.
  3. Standardization: The features are standardized (converted to z-scores) based on the entire training set, contextualizing how "typical" a feature's activation is.
  4. Discretization: Instead of traditional dimensionality reduction, FNE discretizes the standardized values into a {-1, 0, 1} domain. This regularizes the massive 12,416-dimensional space, combating the curse of dimensionality before the vector is mapped to the multimodal space via an affine transformation.

Experimental Setup

The authors evaluated their approach on three standard public benchmarks for image-text matching:

  • Flickr8k: 8,000 images with 5 captions each.
  • Flickr30k: 31,783 images with 5 captions each.
  • MSCOCO: A large-scale dataset with over 82k training images, evaluated on both the widely used 1,000-image and 5,000-image test splits.

Results and Discussion

The authors compared their FNE-equipped model (FN-MME) against a strictly controlled baseline using standard last-layer embeddings (CNN-MME*).

Performance comparison on the MSCOCO dataset (1K and 5K test splits) for image annotation and retrieval tasks.

Key insights from the results include:

  • Multi-scale matters: Features from early and middle layers of a CNN contain useful structural and textural information that aids in matching images to complex, nuanced textual descriptions.
  • Discretization as regularization: Mapping continuous feature activations to a simple ternary state effectively regularizes a massive feature space without requiring lossy dimensionality reduction techniques.
  • Performance Gains: FN-MME consistently outperformed the baseline. On the MSCOCO 1K test split for Image Annotation (Recall@1), FN-MME achieved 47.3% compared to the baseline's 41.2%. On Flickr30k Image Retrieval (Recall@1), FN-MME scored 22.1% vs the baseline's 18.9%.

While FN-MME significantly improved the base pipeline, it did not strictly beat highly specialized, state-of-the-art models (like W2VV for annotation or Fisher Vectors for retrieval), though it closed the gap considerably, especially on image annotation tasks.

Conclusion

The FNE proved to be a superior image representation in this baseline pipeline. Because FNE simply replaces the image feature extraction step, it is highly flexible and can be integrated into more advanced, task-specific architectures. Future work involves integrating FNE into current state-of-the-art pipelines to determine if this multi-scale embedding can establish new benchmarks for the field.

Summary updated

Abstract

The current state-of-the-art for image annotation and image retrieval tasks is obtained through deep neural networks, which combine an image representation and a text representation into a shared embedding space. In this paper we evaluate the impact of using the Full-Network embedding in this setting, replacing the original image representation in a competitive multimodal embedding generation scheme. Unlike the one-layer image embeddings typically used by most approaches, the Full-Network embedding provides a multi-scale representation of images, which results in richer characterizations. To measure the influence of the Full-Network embedding, we evaluate its performance on three different datasets, and compare the results with the original multimodal embedding generation scheme when using a one-layer image embedding, and with the rest of the state-of-the-art. Results for image annotation and image retrieval tasks indicate that the Full-Network embedding is consistently superior to the one-layer embedding. These results motivate the integration of the Full-Network embedding on any multimodal embedding generation scheme, something feasible thanks to the flexibility of the approach.
BibTeX
@inproceedings{vilalta2017full,
  author = {Armand Vilalta and Dario Garcia-Gasulla and Ferran Parés and Jonathan Moreno and Eduard Ayguadé and Jesús Labarta and Ulises Cortés and Toyotaro Suzumura},
  title = {{Full-Network Embedding in a Multimodal Embedding Pipeline}},
  booktitle = {SemDeep@IWCS},
  year = {2017},
}
Figure