Abstract
Traditional Information Retrieval (IR) models often struggle to capture semantic relationships between words due to their reliance on discrete bag-of-words representations. This paper introduces a novel framework that treats text as a "Bag-of-Embedded-Words" (BoEW) and leverages the Fisher Vector (FV) aggregation technique—originally developed for computer vision—to create robust, fixed-length document representations. Our results demonstrate that this approach significantly outperforms traditional topic models like LDA and LSI in both retrieval and clustering tasks.
Introduction
The "Vector Space Model" has long been the cornerstone of Information Retrieval, representing documents as discrete bags-of-words. However, this approach treats words as isolated entities, failing to account for semantic similarities (e.g., "automobile" and "car" are treated as entirely distinct dimensions). While topic models like Latent Dirichlet Allocation (LDA) attempt to bridge this gap by projecting documents into a latent topic space, they are often "coupled" to a fixed vocabulary, making them inflexible to out-of-vocabulary terms.
This paper proposes a framework that maps words into a continuous Euclidean space where distance correlates with semantic similarity. By applying the Fisher Kernel framework, we transform these individual word vectors into a single, high-performing document vector that captures rich semantic information.
Methodology: From Pixels to Prose
The proposed pipeline is inspired by how local image features are aggregated into global image descriptors in computer vision. It consists of two main phases:
- The Bag-of-Embedded-Words (BoEW): Instead of a sparse list of word counts, a document is viewed as a collection of real-valued vectors. These embeddings are generated via Latent Semantic Indexing (LSI) or other word-association measures.
- Fisher Vector (FV) Aggregation: To represent a document, the authors fit a Gaussian Mixture Model (GMM) to the word embedding space. Each Gaussian in the mixture acts as a latent "topic." Rather than simply averaging word vectors, the Fisher Vector calculates the gradient of the document’s words with respect to the GMM parameters. This describes how the words in a specific document deviate from the "average" distribution of the entire corpus, resulting in a high-dimensional, non-linear mapping.
Experimental Setup
The framework was evaluated on two distinct tasks: clustering and information retrieval. For IR, the authors utilized three standard benchmarks from the TREC and CLEF campaigns: ROBUST (TREC), TREC 1&2, and CLEF03.
Key Results and Insights
The experiments revealed that the Fisher Vector approach is highly effective across both tasks:
- Clustering: On the 20 NewsGroups dataset, the FV achieved an Adjusted Rand Index (ARI) of 45.2, comfortably surpassing PLSA (41.0), LDA (40.7), and standard LSI (41.0).
- Information Retrieval: The improvements in IR were even more dramatic. On the CLEF’03 dataset, the FV achieved a Mean Average Precision (MAP) of 23.7%, nearly tripling the performance of the baseline LSI model (9.2%).
As illustrated in the performance plots below, the MAP generally increases as the number of Gaussians ($K$) in the GMM increases, demonstrating that finer-grained topic modeling leads to more discriminative document signatures. Furthermore, the FV model consistently outperforms the LSI baseline across different embedding dimensions ($e$).
A key insight of this work is the "decoupling" of word embeddings from the document model. Unlike LDA, which learns embeddings and topics simultaneously, this method allows for a GMM to be learned on a subset of data and still assign probabilities to "unseen" words, provided they exist in the same continuous embedding space.
Limitations and Future Directions
While the results are promising, the document representation's performance is heavily dependent on the quality of the underlying word embeddings. The authors suggest that "neural" embeddings could yield even better results than the LSI and Yule embeddings used in this study. Additionally, the current model assumes diagonal covariance matrices for the GMMs for computational efficiency; exploring full covariance might capture more complex word relationships. Finally, the framework is naturally suited for multilingual retrieval, as words from different languages could be embedded into a shared space and aggregated using a single universal GMM.
Summary updated