Abstract
Traditional Fisher Vector (FV) representations rely on Gaussian Mixture Models (GMMs), which often fail to capture the heavy-tailed statistics of local image descriptors and word embeddings. This paper proposes a Hybrid Gaussian-Laplacian Mixture Model (HGLMM) that adaptively selects between Gaussian and Laplacian distributions for each feature dimension. Our approach demonstrates superior performance in image annotation, image search, and sentence synthesis across multiple benchmarks, including Flickr8K, Flickr30K, and MS COCO, establishing a new state-of-the-art for FV-based representations at the time of publication.
Introduction: Beyond the Gaussian Assumption
In computer vision, global image representations are frequently constructed by pooling local descriptors like SIFT. The Fisher Vector has long been a preferred method for this pooling, typically derived from the log-likelihood gradients of a GMM. However, local descriptors often exhibit "heavy tails"—distributions where extreme values occur more frequently than predicted by a normal distribution. Recent research suggests that the Laplacian distribution provides a more accurate fit for such data. This paper explores whether replacing GMMs with Laplacian and Hybrid models can enhance performance in cross-modal tasks, such as matching images with natural language descriptions.
Methodology
The core innovation is the derivation of two new mixture models for Fisher Vector encoding:
- Laplacian Mixture Model (LMM): A model based entirely on the Laplacian distribution, utilizing $L_1$ distance metrics rather than the Euclidean ($L_2$) distance used by Gaussians.
- Hybrid Gaussian-Laplacian Mixture Model (HGLMM): A flexible model defined as a weighted geometric mean of Gaussian and Laplacian distributions.
The "Sharp Choice" Mechanism
A key finding in the HGLMM training process is the "sharp choice" behavior during the Maximization step of the Expectation-Maximization (EM) algorithm. Rather than a soft blend, the algorithm makes a binary decision for each dimension of every mixture component, modeling it as either Gaussian or Laplacian based on which fits the data better. This granularity allows the model to capture diverse statistical properties within the same feature set.
For text representation, the authors apply HGLMMs to pool word2vec embeddings into fixed-length sentence vectors. These vectors are then used as inputs for a Recurrent Neural Network (RNN) with LSTM units to perform sentence synthesis.
Experimental Results
The proposed models were evaluated on Pascal1K, Flickr8K, Flickr30K, and MS COCO.
Performance on Flickr Benchmarks
The HGLMM consistently outperformed standard GMMs across tasks. On the Flickr8K dataset, using VGG features, the HGLMM achieved an Image Search Recall@1 of 24.9%, compared to 20.7% for the Mean Vector baseline.
Similarly, on the Flickr30K dataset, the HGLMM reached an Image Annotation Recall@1 of 29.3%, surpassing the standard GMM's 28.4%. These results demonstrate that the HGLMM-based pipeline can outperform several complex deep learning architectures, such as BRNNs and SDT-RNNs.
Key Insights
- Distributional Flexibility: The ability to handle heavy-tailed data is crucial for gradient-based descriptors and semantic embeddings. One dimension of a feature might be normally distributed, while another is heavy-tailed; HGLMM handles this automatically.
- Complementary Information: The best overall results were often achieved by concatenating GMM and HGLMM Fisher Vectors, suggesting that the two models capture distinct, useful information.
- Preprocessing: For Laplacian models, Independent Component Analysis (ICA) proved superior to PCA, as it does not assume underlying Gaussianity.
Conclusion
By relaxing the Gaussian assumption, the Hybrid Gaussian-Laplacian Mixture Model provides a more robust foundation for Fisher Vector representations. The HGLMM's ability to adapt to the specific statistical properties of each feature dimension leads to significant gains in image-text matching tasks. Future work may explore further parametric distributions and specialized normalization techniques for Laplacian-derived vectors.
Summary updated