Full-Network Embedding in a Multimodal Embedding Pipeline

Armand Vilalta, Dario Garcia-Gasulla, Ferran Parés, Eduard Ayguadé, Jesus Labarta, Ulises Cortés, Toyotaro Suzumura

TL;DR

This paper demonstrates that using features from every layer of a deep neural network—rather than just the final layer—significantly boosts performance in multimodal tasks like image annotation and retrieval. By capturing multi-scale visual information through a process called Full-Network Embedding (FNE), the authors achieve consistent improvements across major benchmarks, proving that "richer" image representations lead to better alignment with textual descriptions.

What Problem Does This Paper Address?

In the world of computer vision and natural language processing, a common goal is to create a shared "multimodal" space where images and text can be compared directly. This allows a computer to find a caption for a new image (annotation) or find an image based on a text query (retrieval).

Most current systems use a "one-layer" approach: they take a pre-trained Convolutional Neural Network (CNN), chop off the top, and use the activations from the very last layer as the image's "fingerprint." The authors argue that this creates a bottleneck. A single layer might capture high-level concepts (like "dog") but lose the low-level textures or mid-level parts (like "fur" or "paws") that are distributed throughout the network's earlier layers. The challenge is how to extract and combine these multi-scale features without being overwhelmed by the "curse of dimensionality."

Methods

The authors take a standard multimodal pipeline (originally proposed by Kiros et al.) and replace the standard image encoder with their Full-Network Embedding (FNE).

Figure 1: Overview of the proposed multimodal embedding generation pipeline with the integrated full- network embedding. Elements colored in orange are components modified during the neural network training phase. During testing, only one of the inputs is provided.

The FNE process involves several clever steps to make multi-layer features usable:

  1. Feature Extraction: They extract neural activations from all convolutional and fully-connected layers of a VGG16 network.
  2. Spatial Pooling: To reduce the size of convolutional layers, they apply spatial average pooling to each filter.
  3. Standardization: Every feature is converted into a z-score based on the training set. This tells the system how "atypical" a feature is for a specific image—zero represents average behavior, while high positive or negative values represent significant presence or absence.
  4. Discretization: To handle the massive dimensionality (12,416 features), they map these z-scores to a simple $\{-1, 0, 1\}$ domain. This acts as a powerful regularizer, preserving the "meaning" of the features while making the high-dimensional space easier for the model to learn.

The resulting FNE vector is then mapped into the shared multimodal space using a simple affine transformation, where it is trained to align with text embeddings generated by a Gated Recurrent Unit (GRU) network.

Datasets

The researchers evaluated their approach on three standard benchmarks:

  • Flickr8k: 8,000 images, each with five descriptive captions.
  • Flickr30k: An expansion of Flickr8k containing over 31,000 images of everyday scenes.
  • MSCOCO: A large-scale dataset with over 82,000 training images, widely considered the gold standard for these tasks.

Key Insights and Ideas

  • Multi-scale is Mandatory: The consistent gains show that the "bottleneck" of using only the last CNN layer is real. Earlier layers provide essential context that helps the model distinguish between similar scenes.
  • Discretization over Compression: Instead of using traditional dimensionality reduction like PCA, which can be computationally expensive and lose information, the authors show that simple discretization ($\{-1, 0, 1\}$) is highly effective for regularizing high-dimensional neural features.
  • Plug-and-Play Compatibility: FNE isn't a new architecture; it's a new way to extract features. This means it can theoretically be dropped into almost any existing multimodal pipeline to provide a performance boost.

Key Results

The authors compared their Full-Network approach (FN-MME) against a baseline using the same pipeline but with a standard one-layer embedding (CNN-MME*).

  • Consistent Gains: Across all datasets, FNE outperformed the one-layer baseline. In the Flickr datasets, it improved performance by an average of 3 percentage points.
  • Scaling Up: The benefits were even more pronounced on the larger MSCOCO dataset, showing a roughly 4-point improvement.
  • Annotation Performance: On the MSCOCO 1K test set, FN-MME achieved a Recall@1 (finding the correct caption on the first try) of 47.3%, significantly higher than the one-layer baseline of 41.2%.
Table 3: Results obtained for the MSCOCO dataset. Top part shows results when using 1,000 images as test set, while the bottom shows results when using 5,000 images for test. R@K is Recall@K (high is good). Med r is Median rank (low is good). Best results are shown in bold. † results not in original paper.

Limitations and Future Work

While FNE significantly improved the general pipeline, it did not always beat the absolute "State-of-the-Art" (SotA) models that are highly specialized for a single task. For instance, models like W2VV (specialized for annotation) or Fisher Vectors (specialized for retrieval) still hold the top spots in their respective categories.

The authors note that their next step is to integrate FNE into these specialized SotA pipelines. If the performance boost seen here translates to those models, it could lead to new record-breaking results across the board. Additionally, managing the 12,416-dimensional vector remains a challenge for computational efficiency in very large-scale applications.

Summary updated

Abstract

The current state-of-the-art for image annotation and image retrieval tasks is obtained through deep neural networks, which combine an image representation and a text representation into a shared embedding space. In this paper we evaluate the impact of using the Full-Network embedding in this setting, replacing the original image representation in a competitive multimodal embedding generation scheme. Unlike the one-layer image embeddings typically used by most approaches, the Full-Network embedding provides a multi-scale representation of images, which results in richer characterizations. To measure the influence of the Full-Network embedding, we evaluate its performance on three different datasets, and compare the results with the original multimodal embedding generation scheme when using a one-layer image embedding, and with the rest of the state-of-the-art. Results for image annotation and image retrieval tasks indicate that the Full-Network embedding is consistently superior to the one-layer embedding. These results motivate the integration of the Full-Network embedding on any multimodal embedding generation scheme, something feasible thanks to the flexibility of the approach.
BibTeX
@inproceedings{vilalta2017full,
  author = {Armand Vilalta and Dario Garcia-Gasulla and Ferran Parés and Eduard Ayguadé and Jesus Labarta and Ulises Cortés and Toyotaro Suzumura},
  title = {{Full-Network Embedding in a Multimodal Embedding Pipeline}},
  booktitle = {SemDeep@IWCS},
  year = {2017},
}
Figure