Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

Benjamin Klein, Guy Lev, Gil Sadeh, Lior WolfarXiv 2014 · OpenReview

Abstract

Traditional Fisher Vector (FV) representations rely on Gaussian Mixture Models (GMMs), which often fail to capture the heavy-tailed statistics of local image descriptors and word embeddings. This paper proposes a Hybrid Gaussian-Laplacian Mixture Model (HGLMM) that adaptively selects between Gaussian and Laplacian distributions for each feature dimension. Our approach demonstrates superior performance in image annotation, image search, and sentence synthesis across multiple benchmarks, including Flickr8K, Flickr30K, and MS COCO, establishing a new state-of-the-art for FV-based representations at the time of publication.

Introduction: Beyond the Gaussian Assumption

In computer vision, global image representations are frequently constructed by pooling local descriptors like SIFT. The Fisher Vector has long been a preferred method for this pooling, typically derived from the log-likelihood gradients of a GMM. However, local descriptors often exhibit "heavy tails"—distributions where extreme values occur more frequently than predicted by a normal distribution. Recent research suggests that the Laplacian distribution provides a more accurate fit for such data. This paper explores whether replacing GMMs with Laplacian and Hybrid models can enhance performance in cross-modal tasks, such as matching images with natural language descriptions.

Methodology

The core innovation is the derivation of two new mixture models for Fisher Vector encoding:

  1. Laplacian Mixture Model (LMM): A model based entirely on the Laplacian distribution, utilizing $L_1$ distance metrics rather than the Euclidean ($L_2$) distance used by Gaussians.
  2. Hybrid Gaussian-Laplacian Mixture Model (HGLMM): A flexible model defined as a weighted geometric mean of Gaussian and Laplacian distributions.

The "Sharp Choice" Mechanism

A key finding in the HGLMM training process is the "sharp choice" behavior during the Maximization step of the Expectation-Maximization (EM) algorithm. Rather than a soft blend, the algorithm makes a binary decision for each dimension of every mixture component, modeling it as either Gaussian or Laplacian based on which fits the data better. This granularity allows the model to capture diverse statistical properties within the same feature set.

For text representation, the authors apply HGLMMs to pool word2vec embeddings into fixed-length sentence vectors. These vectors are then used as inputs for a Recurrent Neural Network (RNN) with LSTM units to perform sentence synthesis.

Experimental Results

The proposed models were evaluated on Pascal1K, Flickr8K, Flickr30K, and MS COCO.

Performance on Flickr Benchmarks

The HGLMM consistently outperformed standard GMMs across tasks. On the Flickr8K dataset, using VGG features, the HGLMM achieved an Image Search Recall@1 of 24.9%, compared to 20.7% for the Mean Vector baseline.

Table 1: Performance comparison on the Flickr8K dataset. HGLMM shows consistent improvements over GMM and Mean Vector baselines in both image search and annotation.

Similarly, on the Flickr30K dataset, the HGLMM reached an Image Annotation Recall@1 of 29.3%, surpassing the standard GMM's 28.4%. These results demonstrate that the HGLMM-based pipeline can outperform several complex deep learning architectures, such as BRNNs and SDT-RNNs.

Table 2: Results on the Flickr30K dataset. The HGLMM-based Fisher Vectors achieve state-of-the-art performance compared to contemporary deep learning models.

Key Insights

  • Distributional Flexibility: The ability to handle heavy-tailed data is crucial for gradient-based descriptors and semantic embeddings. One dimension of a feature might be normally distributed, while another is heavy-tailed; HGLMM handles this automatically.
  • Complementary Information: The best overall results were often achieved by concatenating GMM and HGLMM Fisher Vectors, suggesting that the two models capture distinct, useful information.
  • Preprocessing: For Laplacian models, Independent Component Analysis (ICA) proved superior to PCA, as it does not assume underlying Gaussianity.

Conclusion

By relaxing the Gaussian assumption, the Hybrid Gaussian-Laplacian Mixture Model provides a more robust foundation for Fisher Vector representations. The HGLMM's ability to adapt to the specific statistical properties of each feature dimension leads to significant gains in image-text matching tasks. Future work may explore further parametric distributions and specialized normalization techniques for Laplacian-derived vectors.

Summary updated

Abstract

In the traditional object recognition pipeline, descriptors are densely sampled over an image, pooled into a high dimensional non-linear representation and then passed to a classifier. In recent years, Fisher Vectors have proven empirically to be the leading representation for a large variety of applications. The Fisher Vector is typically taken as the gradients of the log-likelihood of descriptors, with respect to the parameters of a Gaussian Mixture Model (GMM). Motivated by the assumption that different distributions should be applied for different datasets, we present two other Mixture Models and derive their Expectation-Maximization and Fisher Vector expressions. The first is a Laplacian Mixture Model (LMM), which is based on the Laplacian distribution. The second Mixture Model presented is a Hybrid Gaussian-Laplacian Mixture Model (HGLMM) which is based on a weighted geometric mean of the Gaussian and Laplacian distribution. An interesting property of the Expectation-Maximization algorithm for the latter is that in the maximization step, each dimension in each component is chosen to be either a Gaussian or a Laplacian. Finally, by using the new Fisher Vectors derived from HGLMMs, we achieve state-of-the-art results for both the image annotation and the image search by a sentence tasks.
BibTeX
@misc{klein2014fisher,
  author = {Benjamin Klein and Guy Lev and Gil Sadeh and Lior Wolf},
  title = {{Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation}},
  year = {2014},
  eprint = {1411.7399},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1411.7399},
}
Figure